From 9fbad9ce92c3ca24831b60bc4d82204e4c195864 Mon Sep 17 00:00:00 2001 From: Scott Hadfield Date: Sat, 26 Sep 2026 14:02:38 -0700 Subject: [PATCH] Add spec documentation and refine lexicons. --- CLAUDE.md | 31 ++- README.md | 2 +- lexicons/README.md | 6 +- .../sharedcomputer/chat/conversationRef.json | 41 ++++ .../network/sharedcomputer/chat/defs.json | 23 ++- .../network/sharedcomputer/chat/info.json | 17 +- .../sharedcomputer/chat/permissions.json | 5 +- .../sharedcomputer/chat/requestSync.json | 25 +++ .../network/sharedcomputer/chat/settings.json | 7 +- specs/README.md | 45 ++++ specs/attachments.md | 102 +++++++++ specs/auth.md | 144 +++++++++++++ specs/browser-store.md | 101 +++++++++ specs/chat-storage.md | 194 ++++++++++++++++++ specs/chat-turns.md | 150 ++++++++++++++ specs/foundation.md | 151 ++++++++++++++ specs/plugins.md | 160 +++++++++++++++ specs/providers.md | 160 +++++++++++++++ specs/search.md | 72 +++++++ specs/sharing.md | 88 ++++++++ specs/titles.md | 80 ++++++++ specs/web-ui.md | 99 +++++++++ 22 files changed, 1672 insertions(+), 31 deletions(-) create mode 100644 lexicons/network/sharedcomputer/chat/conversationRef.json create mode 100644 lexicons/network/sharedcomputer/chat/requestSync.json create mode 100644 specs/README.md create mode 100644 specs/attachments.md create mode 100644 specs/auth.md create mode 100644 specs/browser-store.md create mode 100644 specs/chat-storage.md create mode 100644 specs/chat-turns.md create mode 100644 specs/foundation.md create mode 100644 specs/plugins.md create mode 100644 specs/providers.md create mode 100644 specs/search.md create mode 100644 specs/sharing.md create mode 100644 specs/titles.md create mode 100644 specs/web-ui.md diff --git a/CLAUDE.md b/CLAUDE.md index a61f72c..fb835e6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,21 +2,31 @@ An AI assistant chat app where each conversation is an atproto permissioned space on the user's own PDS. Only the lexicons exist so far. There is no build, test, or app code yet. +## The founding principle + +The user's PDS is the source of truth for every chat. This is the reason the project exists. The server holds no copy of chat content for spaces users: no message tables, no caches that outlive a request, no search index. It reads from the PDS when it needs data, and it keeps only what it needs to operate, such as sessions, sync cursors, reply claims, and encrypted API keys. The local database fallback is a temporary stand-in for a PDS while spaces are in alpha, shaped like a PDS so it can be deleted later. Any design that makes the server a second source of truth, even as a cache, is wrong. + ## Architecture decisions These are settled. Raise a concern before working against any of them. -- Every record belongs to the user, including the assistant's replies. The app never has its own PDS account and never writes records under its own DID. -- One space per conversation, of type `network.sharedcomputer.chat.conversation`. Sharing a chat means changing that space's read policy. -- There is no public HTTP API. The user's PDS is the API: a client can hold a whole conversation by writing records directly. The conventions for that are in `lexicons/README.md`, and the server must honor them exactly. -- Users whose PDS does not support spaces fall back to local storage in SQLite or Postgres. Keep this behind one storage interface so it can be deleted once PDSs support spaces. Local rows store the exact lexicon record JSON under the same space and record keys, so migrating a user to their PDS is a replay. +- Every record belongs to the user, including the assistant's replies. The running app never writes records under its own DID. The one exception is publishing the lexicons, which the maintainers do from a lexicon authority account, outside the app. +- One space per conversation, of type `network.sharedcomputer.chat.conversation`. Sharing a chat means changing that space's read policy. Fallback users cannot share or view shared chats. +- There is no public HTTP API, apart from XRPC procedures defined in our own lexicons, currently `requestSync`. The user's PDS is the API: a client can hold a whole conversation by writing records directly. The conventions for that are in `lexicons/README.md`, and the server must honor them exactly. +- Users whose PDS does not support spaces fall back to local storage in SQLite or Postgres. It sits behind the same PDS-shaped record interface as the spaces backend, so it can be deleted once PDSs support spaces. Local rows store the exact lexicon record JSON under the same space and record keys, so migrating a user to their PDS is a replay. +- Each conversation has an entry record in the user's settings space. Together the entries are the user's chat index, which is how the server and browser find conversations and notice changes with one call per user. +- The browser keeps its own local copy of the user's chats in SQLite compiled to WebAssembly, for the chat list, the open conversation, and search. That copy lives on the user's device, not on the server. - BYO API keys are stored encrypted in the app database, never in records. - Providers, tools, file ingesters, and turn hooks are plugins. Web search, fetch, and image generation are plugins, not core. - The project is meant to be forked. Code refers to lexicon NSIDs only through one generated module, and app name, URLs, OAuth metadata, and admin models come from environment config. ## Planned stack -TypeScript throughout, as a pnpm workspace on Node 22 or newer. Hono server with server-sent events for streaming. Kysely over SQLite and Postgres. Vite and React for the frontend, with no framework on top. Vitest for tests. +TypeScript throughout, as a pnpm workspace on Node 24 LTS, run natively with Node's type stripping and no compile step. Hono server with server-sent events for streaming. Kysely over SQLite and Postgres. Vite and React for the frontend, with no framework on top. Vitest for tests. Provider plugins use the Vercel AI SDK internally, behind our own plugin interface. + +Prefer existing libraries over writing our own, especially the official `@atproto/*` packages for anything protocol related. Dependencies must have licenses compatible with MIT. + +Admins, their global models, and roles are configured through environment config and a config file in phase 1. Roles gate which admin models a user may use. There is no admin UI yet. The web UI must stay barebones with minimal code and styling. A separate designer will build the real interface on top of it. @@ -30,14 +40,15 @@ The web UI must stay barebones with minimal code and styling. A separate designe ## atproto spaces alpha - The spaces SDK is the `@atproto/*` packages at `0.0.0-spaces-alpha-*` versions. Pin exact versions, since breaking changes land without notice. -- These packages only install with pnpm. `@atproto/lex-data` must be overridden to a stable release (`^0.1.7`), because the alpha versions resolve it to a broken build. +- These packages only install with pnpm. Every `@atproto/*` package with an alpha release must be pinned to the exact same alpha version through `pnpm.overrides`. A range like `^0.0.0-spaces-alpha-*` also matches the plain `0.0.0` releases, which were published broken. - Reference material: the permissioned data proposal (`bluesky-social/proposals`, `0016-permissioned-data`), the `permissioned-data-alpha` branch of `bluesky-social/atproto`, and the `bluesky-social/bulletin` example app. ## Server build-out notes -- To see writes it did not make, the server calls `registerNotify` on each conversation space and renews it before it expires. That needs a did:web identity for the app and inbound XRPC routes for `notifyWrite` and `notifySpaceDeleted`, as in bulletin. Notifications are best-effort, so also poll `listRepoOps` from a stored revision per conversation. -- A PDS without spaces rejects the space permissions. Login needs two scope sets and a check for spaces support before the OAuth request. +- To see writes it did not make, the server calls `registerNotify` on each user's settings space, which holds the chat index, and renews it before its one-day expiry. `registerNotify` only accepts a space credential, even for the owner's own space. It also needs a did:web identity for the app and inbound XRPC routes for `notifyWrite` and `notifySpaceDeleted`, as in bulletin. A did:web on a loopback URL cannot be resolved by a PDS, so notifications never arrive in local development. There, sync relies on the safety net, the sync button, and opening a chat. +- A PDS without spaces silently drops space permissions and login still succeeds. Detect spaces support after login from the granted scope, and fall back to local storage when the space permissions are missing. +- The OAuth client must be confidential (`private_key_jwt` with a keyset). Public clients get two-week sessions, and the server needs long-lived sessions to write replies in the background. - Write the conversation's `info` record when creating the space. `listSpaces` only returns spaces the user has written to, so an empty conversation would be invisible. -- The alpha PDS caps a write request at 1 MB. Large tool output, like a fetched page, goes into a blob instead of the record. -- New record types must be added to both the space declaration and the permission set, and every user then has to re-authorize. Batch them. +- The alpha PDS caps a write request at 1 MB, and `putRecord` has no compare-and-swap. Use `createRecord` with an explicit key and catch `RecordAlreadyExists` when a write must not clobber. Large tool output, like a fetched page, goes into a blob instead of the record. +- New record types must be added to both the space declaration and the permission set. Existing sessions pick up changes to the same permission set on token refresh, but a new permission set NSID needs a fresh login. - Blobs in spaces are private on the reference PDS. Verify that on any other PDS implementation before relying on it. diff --git a/README.md b/README.md index ca4eed4..0f5773c 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Phase 1: - Image attachments for vision models - PDF text extraction - Automatic chat titles -- Chat search +- Chat history search - User and admin model configuration Future: diff --git a/lexicons/README.md b/lexicons/README.md index 4bcdbea..7e13f0a 100644 --- a/lexicons/README.md +++ b/lexicons/README.md @@ -7,10 +7,12 @@ SCN Chat's record types use the `network.sharedcomputer.chat` domain. Each conve | Space type | Key | Holds | |---|---|---| | `conversation` | TID | `info`, `message` | -| `settings` | `self` | `preferences` | +| `settings` | `self` | `preferences`, `conversationRef` | Each conversation's URI is `at://{ownerDid}/space/network.sharedcomputer.chat.conversation/{tid}`. Sharing is done via the space's read policy. +Each conversation has a `conversationRef` in the settings space. This is the user's chat index. The title in the conversation's `info` record is authoritative, and the entry contains a copy for the listing. + The system prompt for a conversation is built from the user's custom instructions followed by the conversation system prompt. ## Writing a conversation by hand @@ -18,6 +20,8 @@ The system prompt for a conversation is built from the user's custom instruction A conversation can be held by writing records directly to the PDS without interacting with an appview. - Write a `message` with `role: "user"`, a TID key, and a `generation` object to ask for a reply. Messages without `generation` won't trigger a reply. +- After changing a conversation, put its `conversationRef` with a new `updatedAt`. Its record key is the conversation's space key. The app watches the chat index, so this is how it notices the change. +- A client can call `network.sharedcomputer.chat.requestSync` on the app with service auth from its PDS, to have the app sync immediately. Minting that token needs an `rpc:network.sharedcomputer.chat.requestSync` permission for the app's DID in the client's own OAuth scope. - The reply is written with the key `{userMessageKey}.r{attempt}`, where `attempt` comes from the `generation` object and defaults to 0. - To regenerate, update the user message with `attempt` raised by one. Earlier replies stay as siblings. Order them by attempt number, since `.r10` sorts before `.r2`. - When the app picks up a turn, it first writes the reply with `status: "pending"` and empty content, then updates it when generation ends. The write fails if the key already exists, so only one generation runs per attempt. diff --git a/lexicons/network/sharedcomputer/chat/conversationRef.json b/lexicons/network/sharedcomputer/chat/conversationRef.json new file mode 100644 index 0000000..c906329 --- /dev/null +++ b/lexicons/network/sharedcomputer/chat/conversationRef.json @@ -0,0 +1,41 @@ +{ + "lexicon": 1, + "id": "network.sharedcomputer.chat.conversationRef", + "defs": { + "main": { + "type": "record", + "description": "The conversation index, stored in the user's settings space. Its record key is the conversation's space key. Clients must update it whenever they change the conversation.", + "key": "tid", + "record": { + "type": "object", + "required": ["conversation", "updatedAt"], + "properties": { + "conversation": { + "type": "string", + "format": "space-ref", + "maxLength": 1024, + "description": "The conversation's space URI." + }, + "title": { + "type": "string", + "maxLength": 3000, + "maxGraphemes": 300, + "description": "Copy of the title in the conversation's info record." + }, + "tags": { + "type": "array", + "description": "The user's tags for organizing their chats.", + "maxLength": 32, + "items": { "type": "string", "maxLength": 640, "maxGraphemes": 64 } + }, + "sealed": { + "type": "ref", + "ref": "network.sharedcomputer.chat.defs#encryptedContent", + "description": "When the conversation is encrypted, holds title and tags. Those fields are then left out of the record." + }, + "updatedAt": { "type": "string", "format": "datetime" } + } + } + } + } +} diff --git a/lexicons/network/sharedcomputer/chat/defs.json b/lexicons/network/sharedcomputer/chat/defs.json index 9fa93e3..f5faee2 100644 --- a/lexicons/network/sharedcomputer/chat/defs.json +++ b/lexicons/network/sharedcomputer/chat/defs.json @@ -15,6 +15,11 @@ "id": { "type": "string", "maxLength": 256 } } }, + "providerData": { + "type": "string", + "maxLength": 100000, + "description": "JSON-encoded provider options for this part, such as reasoning signatures. Sent back unchanged when continuing with the same provider." + }, "effort": { "type": "string", "description": "The model's reasoning effort.", @@ -87,18 +92,15 @@ "type": "object", "required": ["text"], "properties": { - "text": { "type": "string", "maxLength": 500000 } + "text": { "type": "string", "maxLength": 500000 }, + "providerData": { "type": "ref", "ref": "#providerData" } } }, "reasoningPart": { "type": "object", "properties": { "text": { "type": "string", "maxLength": 500000 }, - "signature": { - "type": "string", - "maxLength": 100000, - "description": "Token that some providers issue with their reasoning. Sent back unchanged when continuing with the same provider." - } + "providerData": { "type": "ref", "ref": "#providerData" } } }, "imagePart": { @@ -141,7 +143,8 @@ "type": "string", "maxLength": 200000, "description": "JSON-encoded arguments." - } + }, + "providerData": { "type": "ref", "ref": "#providerData" } } }, "toolResultPart": { @@ -154,6 +157,12 @@ "maxLength": 500000, "description": "Tool output as text or JSON-encoded data." }, + "outputBlob": { + "type": "blob", + "accept": ["*/*"], + "maxSize": 50000000, + "description": "Used when the output is too long for the output record. Output will contain a summary in that case." + }, "isError": { "type": "boolean", "default": false } } }, diff --git a/lexicons/network/sharedcomputer/chat/info.json b/lexicons/network/sharedcomputer/chat/info.json index 4e97b79..856ddc6 100644 --- a/lexicons/network/sharedcomputer/chat/info.json +++ b/lexicons/network/sharedcomputer/chat/info.json @@ -4,29 +4,28 @@ "defs": { "main": { "type": "record", - "description": "Title, tags, and defaults for a conversation.", + "description": "Title and system prompt for a conversation.", "key": "literal:self", "record": { "type": "object", "required": ["createdAt"], "properties": { - "title": { "type": "string", "maxLength": 3000, "maxGraphemes": 300 }, + "title": { + "type": "string", + "maxLength": 3000, + "maxGraphemes": 300, + "description": "The chat title. This is the authoritative record, conversationRef holds a copy for the chat listing." + }, "titleSource": { "type": "string", "description": "How the title was created: by the user or auto-generated. Apps must not replace a title set by the user.", "knownValues": ["user", "generated"] }, - "tags": { - "type": "array", - "maxLength": 32, - "items": { "type": "string", "maxLength": 640, "maxGraphemes": 64 }, - "description": "Taxonomy terms to label a conversation with." - }, "systemPrompt": { "type": "string", "maxLength": 100000 }, "sealed": { "type": "ref", "ref": "network.sharedcomputer.chat.defs#encryptedContent", - "description": "When encrypted, holds title, tags, and the systemPrompt. Those fields are then left out of the record." + "description": "When encrypted, holds title and the systemPrompt. Those fields are then left out of the record." }, "createdAt": { "type": "string", "format": "datetime" } } diff --git a/lexicons/network/sharedcomputer/chat/permissions.json b/lexicons/network/sharedcomputer/chat/permissions.json index 17c06a3..fc38d56 100644 --- a/lexicons/network/sharedcomputer/chat/permissions.json +++ b/lexicons/network/sharedcomputer/chat/permissions.json @@ -32,7 +32,10 @@ "spaceType": "network.sharedcomputer.chat.settings", "authority": "self", "skey": "self", - "collection": ["network.sharedcomputer.chat.preferences"], + "collection": [ + "network.sharedcomputer.chat.preferences", + "network.sharedcomputer.chat.conversationRef" + ], "action": ["read", "create", "update", "delete"], "manage": ["create", "update", "delete"] }, diff --git a/lexicons/network/sharedcomputer/chat/requestSync.json b/lexicons/network/sharedcomputer/chat/requestSync.json new file mode 100644 index 0000000..3bf81ce --- /dev/null +++ b/lexicons/network/sharedcomputer/chat/requestSync.json @@ -0,0 +1,25 @@ +{ + "lexicon": 1, + "id": "network.sharedcomputer.chat.requestSync", + "defs": { + "main": { + "type": "procedure", + "description": "Trigger a sync of the caller's chats from their PDS, and reply to any pending generation requests.", + "input": { + "encoding": "application/json", + "schema": { + "type": "object", + "properties": { + "conversation": { + "type": "string", + "format": "space-ref", + "maxLength": 1024, + "description": "A conversation's space URI." + } + } + } + }, + "errors": [{ "name": "UnknownConversation" }] + } + } +} diff --git a/lexicons/network/sharedcomputer/chat/settings.json b/lexicons/network/sharedcomputer/chat/settings.json index b73577c..ba0445a 100644 --- a/lexicons/network/sharedcomputer/chat/settings.json +++ b/lexicons/network/sharedcomputer/chat/settings.json @@ -4,10 +4,13 @@ "defs": { "main": { "type": "space", - "description": "The user's personal chat settings.", + "description": "The user's chat preferences and conversation index.", "key": "literal:self", "name": "Chat settings", - "collections": ["network.sharedcomputer.chat.preferences"] + "collections": [ + "network.sharedcomputer.chat.preferences", + "network.sharedcomputer.chat.conversationRef" + ] } } } diff --git a/specs/README.md b/specs/README.md new file mode 100644 index 0000000..a22fd4c --- /dev/null +++ b/specs/README.md @@ -0,0 +1,45 @@ +# Specs + +Each file here describes one feature of SCN Chat: what it does, why it exists, how it is built, what it deliberately leaves out, and how to tell it works. A spec is written and agreed on before the feature is built, and kept current as the feature changes. + +Specs are the source of truth for behavior. When code and spec disagree, one of them is wrong, and the fix is to bring them back in line rather than let them drift. A spec should be detailed enough that someone, or an LLM, could rebuild the feature from it plus the project's `CLAUDE.md`. + +These files are all AI generated and maintained. + +The overview at the end of this file lists every spec and whether it is active, planned, or done. + +## Sections + +- **Summary**: what the feature does, for someone meeting it for the first time. +- **Motivation**: the problem it solves, with enough context to make good judgment calls. +- **Design**: the technical approach, data shapes, and where the code lives. +- **Scope Boundaries**: what the feature does not do. +- **Edge Cases and Decisions**: a running log of non-obvious choices. +- **Acceptance Criteria**: checks that each become at least one test. +- **Files**: the code that implements the feature, used to spot specs that have fallen behind the code. + +## Overview of Feature Specs + +### Active + +### Planned + +Phase 1, initial build-out, in build order: + +- [[foundation]] +- [[plugins]] +- [[auth]] +- [[chat-storage]] +- [[providers]] +- [[chat-turns]] +- [[attachments]] +- [[titles]] +- [[sharing]] +- [[browser-store]] +- [[web-ui]] + +Phase 1, after the initial build-out: + +- [[search]] + +### Completed diff --git a/specs/attachments.md b/specs/attachments.md new file mode 100644 index 0000000..aaf3eaa --- /dev/null +++ b/specs/attachments.md @@ -0,0 +1,102 @@ +# Attachments + +## Summary + +Users can attach images and PDFs to a message. Images are sent to vision-capable models as images. PDFs go through a file ingester that extracts their text, and the text is sent to the model. Attachments are blobs referenced from the message record: on the user's PDS for spaces users, and in a local blob store for fallback users. Extracted text is saved as its own blob, referenced from the file part, so a PDF is only processed once. PDF text extraction ships as the first ingester plugin, `@scn-chat/plugin-pdf-text`. + +## Motivation + +Vision and PDF support were on the original phase 1 list. Keeping the file bytes and extracted text as blobs in the user's own repo keeps attachments under the user's control like the rest of the conversation. Making extraction a plugin means OCR, Word documents, or other formats can be added later without core changes. + +## Design + +### Upload flow + +1. The web UI sends the file to `POST /api/attachments` as the raw request body, with its `Content-Type` and an `X-Filename` header. Attachments are uploaded before the message is sent, so the user sees progress and errors early. +2. The server checks the MIME type and size. Images must be PNG, JPEG, WebP, or GIF, up to 20 MB. Other files are accepted when an ingester matches their type, up to 50 MB. These limits match the lexicon. The PDS may enforce a lower limit, and its error is passed through. +3. The server stores the bytes through the blob store below and gets back an atproto blob reference. +4. For non-image files, the matching ingester extracts text. The text is stored as a second `text/plain` blob. +5. The response carries the part to put in the message: an `imagePart`, or a `filePart` with `name` and `extracted: { text, method }`. + +`GET /api/conversations/:skey/blobs/:cid` serves one of the signed-in owner's blobs, with its MIME type, `X-Content-Type-Options: nosniff`, and `Content-Disposition: attachment` for anything but images. For spaces users it calls `com.atproto.space.getBlob` for the conversation, and the PDS refuses any blob not referenced from a record in that space. The server passes that refusal on as 404, with no check of its own. For fallback users it serves the file from the owner's blob directory, or 404 when there is none. Shared conversations serve blobs through the sharing spec's route. + +The web UI includes the returned parts in the message it sends. The server does not track which blobs a user uploaded. For spaces users, the PDS rejects a record that references a blob the account never uploaded, and for fallback users a blob either exists in the owner's directory or does not. + +### Blob store + +`apps/server/src/blobs/` has one interface with two implementations, chosen by the user's storage mode: + +```ts +interface BlobStore { + put(user, bytes: Uint8Array, mimeType: string): Promise + get(user, conversation, ref: BlobRef): Promise +} +``` + +- **Spaces users.** `put` calls `com.atproto.repo.uploadBlob` on the user's PDS with their OAuth session, which needs the blob permission in the permission set. `get` calls `com.atproto.space.getBlob` for the conversation. +- **Fallback users.** Bytes are written under `DATA_DIR/blobs//`, one directory per owner, with the CID computed the way a PDS computes it: CIDv1, raw codec, SHA-256. The same bytes therefore get the same CID if the user later moves to a spaces PDS, so their records' blob references stay valid after migration. + +A PDS keeps an uploaded blob only if a record references it within a time window. An attachment that is never sent is cleaned up by the PDS for spaces users, and by a daily sweep of unreferenced files older than a day for fallback users. + +Blobs are not cached on the server. A turn fetches the images it sends to the model from the blob store each time, and holds them in memory only for that turn. + +### Ingesters + +In `@scn-chat/plugin-api`: + +```ts +interface Ingester { + id: string + accepts: string[] // MIME types, e.g. ['application/pdf'] + priority?: number // higher wins when several match, default 0 + method: 'text' | 'ocr' // written to extracted.method + ingest(file: { bytes: Uint8Array; mimeType: string; name?: string }): Promise<{ text: string }> +} +``` + +The highest-priority ingester that accepts the MIME type runs. An OCR plugin can therefore register as a fallback for scanned PDFs, or take over PDFs entirely with a higher priority. If no ingester accepts a non-image type, the upload is refused with 415. + +`@scn-chat/plugin-pdf-text` uses `unpdf` to extract the text of every page, joined with page breaks. A PDF with no extractable text, such as a scan, fails with a clear message saying the PDF has no text layer. + +### Sending attachments to the model + +When the chat-turns spec builds the model's messages: + +- An `imagePart` becomes an AI SDK file part with the image bytes and MIME type, if the resolved model has the vision capability. If it does not, the turn fails before calling the model with an error saying the model cannot read images. The web UI prevents this by disabling image attachments for such models. +- A `filePart` with extracted text becomes a text part: the file name as a heading, followed by the extracted text. +- A `filePart` without extracted text is left out, with a note in the text telling the model a file was attached but could not be read. + +## Scope Boundaries + +- No OCR. That is a future plugin. +- No sending PDFs to models as native PDF input. +- No attachments on assistant messages. Image generation is a future plugin. +- No audio or video. +- No attachments in encrypted conversations. +- No limit on how much extracted text is sent, beyond the model's own context limit. + +## Edge Cases and Decisions + +- Extraction runs at upload time, not at send time, so a failure shows up before the user sends the message. +- Fallback-mode blobs use the same CID scheme as a PDS, so migration to a spaces PDS keeps blob references valid. +- Spaces users' blobs are fetched from their PDS on every turn that needs them, with no server-side cache, keeping the PDS the only store of their attachments. + +## Acceptance Criteria + +- [ ] Uploading a PNG returns an `imagePart` whose blob can be fetched back through the conversation's blob route once the message is sent. +- [ ] The blob route returns 404 for a CID not referenced in the conversation, and for another user's conversation. +- [ ] Uploading a PDF with text returns a `filePart` with extracted text stored as a `text/plain` blob, with method `text`. +- [ ] A PDF without a text layer fails with a message saying it has no text. +- [ ] An unsupported MIME type is refused with 415. +- [ ] Files over the size limits are refused with 413. +- [ ] Fallback-mode blobs get the same CID a PDS would compute for the same bytes. +- [ ] When several ingesters accept a type, the highest priority runs. +- [ ] An image sent to a model without vision fails the turn before any model call. +- [ ] Extracted text reaches the model as a text part headed by the file name. +- [ ] The blob route returns the PDS's refusal as 404 for a blob not referenced in the conversation, without reading the conversation's records. +- [ ] No blob is written to server storage for a spaces user. +- [ ] The fallback sweep removes unreferenced blobs older than a day and keeps referenced ones. + +## Files + +- (to be populated during implementation) diff --git a/specs/auth.md b/specs/auth.md new file mode 100644 index 0000000..f5ecaaa --- /dev/null +++ b/specs/auth.md @@ -0,0 +1,144 @@ +# Auth + +## Summary + +Users sign in with their atproto account through atproto OAuth, using `@atproto/oauth-client-node`. The server is a confidential OAuth client, so it holds long-lived sessions and can write replies to the user's PDS in the background. After login, the server checks which permissions the PDS actually granted. If the space permissions are there, the account uses spaces storage. If not, because the PDS does not support spaces, the account falls back to local storage. The browser holds only an opaque session cookie, and all PDS calls happen on the server. + +## Motivation + +An atproto account is the only identity SCN Chat uses, for spaces and fallback users alike. OAuth is the standard way for an app to act for a user. The server needs sessions that survive for months, because a reply can be triggered by a record written from another client while the user is away. atproto only grants long sessions (up to two years, refreshed every three months) to confidential clients. Public clients are capped at two weeks. + +## Design + +### OAuth client + +`apps/server/src/auth/client.ts` creates one `NodeOAuthClient`: + +- **Production.** A confidential client with `token_endpoint_auth_method: "private_key_jwt"`, `dpop_bound_access_tokens: true`, and a keyset of ES256 keys from `@atproto/jwk-jose`. The client ID is `${PUBLIC_URL}/oauth-client-metadata.json`. The server publishes the metadata at that path and the public keys at `/oauth/jwks.json`. `PUBLIC_URL` must be HTTPS. +- **Development.** When `PUBLIC_URL` is a loopback address, the server uses `buildAtprotoLoopbackClientMetadata` instead. Loopback clients are always public, so development sessions last two weeks at most. + +The OAuth state and session stores are `oauth_state` and `oauth_session` tables, each with a key, a JSON value, and an updated timestamp. Sessions are keyed by DID. + +### Scope + +The requested scope is `atproto include:network.sharedcomputer.chat.permissions`, with the permission set NSID taken from the lexicon module, never typed as a literal. The permission set grants the conversation, shared-read, settings, and blob permissions. The PDS resolves it through the lexicon's DNS record, so the lexicons must be published under `sharedcomputer.network` before anyone can log in. A fork publishes its own lexicons under its own domain. + +For local development against a PDS that cannot resolve the published permission set, `OAUTH_SCOPE_MODE=raw` requests the equivalent raw scopes instead, built from the same lexicon module: `space:` scopes for the conversation, shared-read, and settings permissions with every collection listed, plus `blob:*/*`. The default is `permission-set`, and production startup refuses `raw`. + +### Login + +1. The web UI sends the user to `GET /oauth/login?identifier=`. +2. The server calls `client.authorize(identifier, { scope })` and redirects to the returned URL. +3. `GET /oauth/callback` calls `client.callback(params)` and gets an `OAuthSession`. +4. The server reads the granted scope with `session.getTokenInfo()`. Using `ScopePermissions` from `@atproto/oauth-scopes`, it checks whether the scope allows creating conversation spaces for the user's own DID. +5. It upserts the `account` row and creates a web session (below), then redirects to `/`. + +A PDS without spaces drops space scopes silently instead of refusing them, so login always succeeds. The check in step 4 is how the server tells the two kinds of PDS apart. + +### Storage mode + +The `account` table holds the DID, current handle, PDS URL, storage mode (`space` or `local`), the `background_sync` setting from the chat-storage spec (default true), and creation, last-login, and last-activity times. + +- A new account whose granted scope allows spaces gets `space`. Otherwise it gets `local`. +- An existing `local` account stays `local` even if its PDS gains spaces support. Moving data to the PDS is a future migration feature. +- An existing `space` account whose granted scope no longer allows spaces fails login with a message explaining that its chats live in spaces the PDS no longer supports. It does not switch to local silently, which would hide the user's existing chats. + +### Web sessions + +The browser gets an `scn_session` cookie holding 32 random bytes, base64url encoded. It is `HttpOnly`, `SameSite=Lax`, `Path=/`, and `Secure` whenever `PUBLIC_URL` is HTTPS. The `web_session` table stores the token's SHA-256 hash, the DID, and creation and expiry times. Sessions last `SESSION_TTL_DAYS`, default 30, and are extended when used in the last half of their life. + +Hono middleware resolves the cookie to the account on every `/api` request. Routes that need a user return 401 without one. + +Every non-GET `/api` request must carry an `Origin` header matching `PUBLIC_URL`, or it is refused with 403. Together with `SameSite=Lax`, that blocks cross-site request forgery. + +### Acting for a user + +Other specs get a PDS client for a user through one function: + +```ts +getPdsClient(did): Promise +``` + +It calls `client.restore(did)` and wraps the session in `@atproto/lex-client`'s `Client`. Token refresh happens inside the OAuth library. If the session is gone or refresh fails, it throws `SessionExpired`. The web UI answers that by sending the user to log in again, and a background turn records it as the reply's error. + +### Roles + +Roles decide what a user may do beyond using their own API keys. In phase 1 they only gate admin models, as the providers spec describes. + +Roles are defined in `scn-chat.config.ts`: + +```ts +roles: { + staff: ['did:plc:abc...', 'did:plc:def...'], + beta: ['did:plc:ghi...'], +} +``` + +- Every signed-in user has the implicit `user` role. The name `user` cannot be defined in the config. +- Role names match `^[a-z][a-z0-9-]*$`, and every entry must be a valid DID. Startup fails otherwise, naming the role or entry. +- `rolesFor(did)` returns `user` plus every configured role listing the DID. Roles are read from the config on each call, so they apply without a new login once the server restarts with a changed config. + +### Logout + +`POST /api/logout` deletes the web session and clears the cookie. It does not revoke the OAuth session, since background replies still need it. `POST /api/logout?everywhere=1` also calls `client.revoke(did)` and deletes all of that user's web sessions. + +### Endpoints + +| Route | Purpose | +|---|---| +| `GET /oauth-client-metadata.json` | OAuth client metadata | +| `GET /oauth/jwks.json` | Public signing keys | +| `GET /oauth/login` | Start login | +| `GET /oauth/callback` | Finish login | +| `GET /api/me` | The signed-in user's DID, handle, storage mode, and roles | +| `POST /api/logout` | Sign out | + +### Configuration + +| Variable | Purpose | +|---|---| +| `OAUTH_PRIVATE_KEYS` | JSON array of ES256 private JWKs, each with a `kid`. Required in production. | +| `OAUTH_SCOPE_MODE` | `permission-set` (default) or `raw`, for local development only. | +| `SESSION_TTL_DAYS` | Web session lifetime. Default 30. | + +A `pnpm keys` script prints a fresh `OAUTH_PRIVATE_KEYS` value and a `SECRET_KEY` for the providers spec. + +## Scope Boundaries + +- No accounts outside atproto, and no passwords. +- No admin role or admin screens in phase 1. Admin configuration lives in the config file. +- No roles stored in the database, and no UI for assigning them. +- No migration of `local` accounts to spaces. +- No revoking the OAuth session on ordinary logout. + +## Edge Cases and Decisions + +- Login depends on the published permission set. An unpublished or unresolvable permission set makes every login fail with `invalid_scope`, and the login page shows that error. +- A spaces account that loses spaces support fails login loudly instead of falling back to local storage. +- Loopback development uses a public client, because atproto OAuth does not allow confidential loopback clients. +- **To verify during implementation:** that a PDS without spaces accepts the permission set and skips its space permissions, as the research into the alpha source suggests. + +## Acceptance Criteria + +- [ ] Client metadata is served at the client ID URL, with `private_key_jwt` and the public keys at `/oauth/jwks.json`. +- [ ] A loopback `PUBLIC_URL` uses loopback client metadata. +- [ ] Production startup fails without valid `OAUTH_PRIVATE_KEYS` or with a non-HTTPS `PUBLIC_URL`. +- [ ] The requested scope includes the permission set, with its NSID taken from the lexicon module. +- [ ] An `invalid_scope` error from the PDS is shown on the login page. +- [ ] `OAUTH_SCOPE_MODE=raw` requests raw space and blob scopes equivalent to the permission set, and production startup refuses it. +- [ ] A new account whose granted scope allows spaces gets storage mode `space`. +- [ ] A new account whose granted scope lacks spaces gets storage mode `local`. +- [ ] A `space` account whose granted scope lacks spaces fails login with an explanation. +- [ ] A `local` account stays `local` when its scope later allows spaces. +- [ ] The session cookie is HttpOnly and SameSite=Lax, and only its hash is stored. +- [ ] `/api` routes return 401 without a valid session and 403 for a non-GET request with a foreign `Origin`. +- [ ] Sessions expire after the TTL and are extended when used in the second half of it. +- [ ] `getPdsClient` throws `SessionExpired` when the OAuth session cannot be restored. +- [ ] Logout clears the web session and keeps the OAuth session, and logout everywhere revokes it. +- [ ] `rolesFor` returns `user` plus every configured role that lists the DID. +- [ ] A config defining a role named `user`, an invalid role name, or an invalid DID fails startup, naming it. +- [ ] `GET /api/me` includes the user's roles. + +## Files + +- (to be populated during implementation) diff --git a/specs/browser-store.md b/specs/browser-store.md new file mode 100644 index 0000000..14eb66d --- /dev/null +++ b/specs/browser-store.md @@ -0,0 +1,101 @@ +# Browser Store + +## Summary + +The web app keeps a local copy of the signed-in user's chats in the browser, in SQLite compiled to WebAssembly (`@sqlite.org/sqlite-wasm`), stored in the browser's origin private file system. The chat list, open conversations, and search all read from this local database. It syncs with the user's PDS through the server's pass-through routes, fetching only changes since the revisions it last saw. The copy lives on the user's device, so the server never holds chat content, and the UI stays fast without a server cache. + +## Motivation + +The PDS is the source of truth, and the server keeps no copy of chat content. Reading everything from the PDS on every screen would be slow, and full-text search over hundreds of conversations is impossible without an index. Keeping the copy and the index on the user's own device gives both, while leaving the server stateless about content. SQLite with FTS5 gives real search and a familiar SQL model in the browser. + +## Design + +### Runtime + +- `@sqlite.org/sqlite-wasm` runs in a dedicated Web Worker, using the `opfs-sahpool` VFS. That VFS persists to the origin private file system without the cross-origin isolation headers the other OPFS mode needs. +- The UI talks to the worker through `comlink`, which exposes the worker's functions as async calls. +- There is one database file per account, named from a hash of the DID, so accounts on a shared browser stay separate. +- Signing out deletes that account's database file. So does any 401 from the server, which means the session has lapsed, so chats are not left on a shared device after an expired session. The data is on the PDS, so a later sign-in resyncs it. +- `opfs-sahpool` allows only one open connection per database, so only one tab holds the store at a time, and it moves to whichever tab the user is working in: + - When a tab gains focus, it requests a Web Lock named after the database with `steal: true`, and posts a release request on a `BroadcastChannel`. + - The tab losing the lock finishes any running transaction, closes its database with SQLite's `pauseVfs`, and keeps showing what it last rendered. Reply streams still reach it, since they come from the server. It reopens the store when it is focused again. + - The focused tab retries opening the database for up to three seconds, since the old tab may need a moment to release its file handles. + - If the open still fails, most likely because the browser has frozen the other tab and it cannot release its handles, the tab shows a banner saying SCN Chat is busy in another tab, with a "use here" button that retries. It keeps showing what it last rendered meanwhile. + - Opening the database either succeeds or throws, so a tab always knows whether it holds the store, and two tabs never hold it at once. +- If the origin private file system is unavailable, as in some private browsing modes, the worker uses an in-memory database and logs a warning. Everything works, but nothing persists between sessions. + +### Schema + +| Table | Columns | +|---|---| +| `conversation` | `skey` (primary key), `uri`, `title`, `tags_json`, `updated_at`, `rev` (the last conversation revision fetched, null if never fetched) | +| `message` | `conversation_skey`, `author_did`, `rkey`, `role`, `parent`, `status`, `record_json`, `text`, `created_at`. Primary key is the first three. | +| `info` | `conversation_skey` (primary key), `record_json` | +| `meta` | `key` (primary key), `value`. Holds the index revision and the schema version. | + +`text` is the concatenated text parts of a message, used by search. The schema version is checked on open. A mismatch deletes and rebuilds the database, since everything in it can be fetched again. + +### Sync + +- **The index.** On start, and whenever the server's event stream says the index changed, the worker calls `GET /api/conversations?since=`. It applies the returned entries and deletions to `conversation`, and stores the new revision. The first call has no revision and returns the whole index. +- **Conversations.** A conversation is fetched with `GET /api/conversations/:skey?since=`, applying changed records and deletions and storing the new revision. The first fetch of a conversation has no revision and returns everything. +- **Opening a conversation** fetches its changes before showing it, and shows the local copy immediately while that runs. +- **Background download.** After the index syncs, the worker fetches every conversation it has never fetched, newest first, two at a time, so search eventually covers the whole history. A conversation whose `updated_at` is newer than when it was last fetched is refetched the same way. +- **Live changes.** The worker subscribes to `GET /api/events`. A `conversation-changed` event fetches that conversation's changes, and an `index-changed` event syncs the index. +- **Own writes.** After the UI writes through the server, the worker fetches the affected conversation's changes, so the local copy matches the PDS. +- **Reconciling.** At startup for the index, and when a conversation is opened, the worker fetches the matching keys route from the chat-storage spec. It compares every collection, key, and CID with its own rows, fetches records it lacks or holds at a different CID, and deletes rows the PDS no longer has. That repairs any gap in the revision-based fetches without downloading content it already has. + +Streaming replies are shown from the turn's stream, as the chat-turns spec describes. The finished record arrives through a normal changes fetch. + +### Worker API + +```ts +listConversations(): Promise +getConversation(skey): Promise<{ info; messages }> +refreshConversation(skey): Promise +syncIndex(): Promise +search(query, opts): Promise // see the search spec +reset(): Promise // delete and rebuild the database +``` + +Changes are announced to the UI through a subscription, so screens rerender when the data under them changes. + +## Scope Boundaries + +- No writes to the local database from the UI. All writes go through the server to the PDS, and the local copy follows. +- No offline writing or queueing. +- No sync between devices other than through the PDS. +- No local copy of shared conversations owned by other people. +- No encryption of the local database beyond what the browser provides. + +## Edge Cases and Decisions + +- The local database is disposable. A schema change or a reset rebuilds it from the PDS, so it never needs migrations. +- Signing out deletes the local copy, trading a resync on next sign-in for not leaving chats on a shared device. +- Search only covers conversations the background download has fetched, so a new device's search fills in over the first minutes. +- `opfs-sahpool` avoids the cross-origin isolation headers, which would break embedding and some third-party resources. The cost is one connection at a time, handled by moving the store to the focused tab. +- The handover cannot be guaranteed, because a frozen background tab cannot release its file handles. The banner covers that case. + +## Acceptance Criteria + +- [ ] The first sync fetches the whole index and stores its revision. +- [ ] Later syncs fetch only changes since the stored revision, including deletions. +- [ ] Opening a conversation shows the local copy, then applies fetched changes. +- [ ] The background download fetches every unfetched conversation, newest first, at most two at a time. +- [ ] A `conversation-changed` event fetches that conversation's changes. +- [ ] After a write through the server, the local copy shows the written record. +- [ ] Two accounts in the same browser get separate databases. +- [ ] Signing out deletes the account's database. +- [ ] A 401 from the server deletes the account's database. +- [ ] Focusing another tab moves the store to it without any notice. +- [ ] The tab that lost the store keeps its screen, and reopens the store when focused again. +- [ ] When the store cannot be opened within three seconds, the tab shows the busy banner, and "use here" retries. +- [ ] Two tabs never hold the store at once. +- [ ] Reconciling fetches missing or changed records and deletes records the PDS no longer has. +- [ ] A schema version mismatch rebuilds the database. +- [ ] Without the origin private file system, the store works in memory. +- [ ] The database survives a page reload. + +## Files + +- (to be populated during implementation) diff --git a/specs/chat-storage.md b/specs/chat-storage.md new file mode 100644 index 0000000..bae395b --- /dev/null +++ b/specs/chat-storage.md @@ -0,0 +1,194 @@ +# Chat Storage + +## Summary + +Chat storage reads and writes a user's conversations, messages, chat index, and preferences. The user's PDS is the source of truth, and the server keeps no copy of chat content. Everything goes through a `RecordStore` interface shaped like a PDS, with two backends: `SpaceRecordStore` calls the user's PDS, and `LocalRecordStore` emulates a PDS in the app database for fallback users. A `ChatService` on top implements chat operations, such as creating a conversation or writing a message, and keeps the chat index in step. Reads pass straight through to the PDS on every request. The server also watches each user's chat index for changes, so a conversation written from any client can get a reply. That watching keeps only cursors, never content. + +## Motivation + +The founding principle of SCN Chat is that the user's PDS is the source of truth. The server may read and write chat data on the user's behalf, but must never become a second store of it. Shaping the fallback backend like a PDS keeps the rest of the code identical for both kinds of user, and makes the fallback easy to delete later. Watching one chat index per user, instead of every conversation, keeps sync cost proportional to users rather than to conversations. + +## Design + +### RecordStore + +```ts +interface RecordStore { + createSpace(type: Nsid, skey?: string): Promise // throws SpaceExists + deleteSpace(space: SpaceUri): Promise + getRecord(space, collection, rkey): Promise<{ value; cid } | null> + listRecords(space, collection, opts?: { cursor?; limit? }): Promise<{ records: { rkey; value; cid }[]; cursor? }> + createRecord(space, collection, rkey, value): Promise<{ cid }> // throws RecordExists + putRecord(space, collection, rkey, value): Promise<{ cid }> + deleteRecord(space, collection, rkey): Promise + listOps(space, since?: string): Promise<{ ops: Op[]; rev: string }> // pages internally +} +``` + +`getRecordStore(did)` returns the backend for the account's storage mode. Every write validates the record with the generated lexicon validators first, and throws `InvalidRecord` with the validation message on failure. + +**SpaceRecordStore** calls the user's PDS through `getPdsClient(did)`: + +- `createSpace` calls `com.atproto.simplespace.createSpace`. Conversations use `readPolicy` and `writePolicy` `#memberListPolicy` and `appAccess` `#open`. `SpaceAlreadyExists` becomes `SpaceExists`. +- Record methods call `com.atproto.space.getRecord`, `listRecords`, `createRecord`, `putRecord`, and `deleteRecord`, with `repo` set to the user's DID and `validate: false`, since our own validation already ran. `RecordAlreadyExists` becomes `RecordExists`. +- `listOps` calls `com.atproto.space.listRepoOps` for the user's own repo and pages until done. + +**LocalRecordStore** stores records in three tables: + +| Table | Columns | +|---|---| +| `local_space` | `uri` (primary key), `owner_did`, `type`, `skey`, `created_at` | +| `local_record` | `space_uri`, `collection`, `rkey`, `value_json`, `cid`, `updated_at`. Primary key is the first three. | +| `local_op` | `seq` (auto-increment primary key), `space_uri`, `collection`, `rkey`, `cid` (null for deletes), `created_at` | + +Local space URIs use the same format as spaces, and record CIDs are computed the way a PDS computes them, with the atproto data libraries. A later migration to a PDS is then a replay of `local_record` with every URI, key, and CID unchanged. `listOps` returns `local_op` rows after the given revision, where the revision is the zero-padded `seq`. + +### ChatService + +`ChatService` implements chat operations for one user, over their `RecordStore`: + +- **Settings space.** At login, it creates the settings space with key `self`, `readPolicy` and `writePolicy` `#memberListPolicy`, and `appAccess` `#open`, treating `SpaceExists` as success. +- **Creating a conversation.** It creates the conversation space, writes the `info` record at key `self`, then writes the conversation's `conversationRef` in the settings space, keyed by the conversation's key, with `updatedAt`. +- **Writing a message.** `createMessage` and `putMessage` write the message, then update the conversation's `conversationRef` with a new `updatedAt`. +- **Updating info.** `updateInfo` reads the info record, applies the patch, and writes it with `putRecord`. With `unlessUserTitled`, it skips the write when the stored `titleSource` is `user`. A title change is copied into the `conversationRef`, since the info record's title is the authoritative one. +- **Tags.** Tags live only in the `conversationRef`, in the owner's private settings space. `setTags` rewrites the entry with the new tags. +- **Deleting a conversation.** It deletes the space, then deletes the `conversationRef`. +- **Preferences.** They are read and written at key `self` in the settings space. + +The index update is a second write, so it can fail after the message write succeeds. A failure is logged as a warning, and discovery repairs the index later. `putRecord` has no compare-and-swap, so two clients writing the same conversation at once can overwrite each other's info or index entry. That is a risk the user accepts by writing from several clients. + +### Reads + +Reads come from the `RecordStore` on every request, with no server-side cache: + +- `listConversations(did, since?)` lists every `conversationRef` in the settings space, newest `updatedAt` first. With `since`, it returns only the entries changed after that revision, from `listOps`, including deletions. Either way it returns the settings space's current revision for the next call. +- `getConversation(did, skey, since?)` returns the info record and every message. With `since`, it returns only the records changed after that revision. Either way it returns the conversation's current revision. + +The browser keeps its own copy of what it reads, as the browser-store spec describes, and passes the revisions back to fetch only changes. + +### Sync + +Sync lets the server notice changes it did not make, so chat-turns can answer generation requests written from any client. It applies only to spaces users. Fallback users can only write through the server, which starts turns directly. + +The server stores cursors and registration state in a `sync_state` table: space URI (primary key), owner DID, `last_rev`, `registered_until`, `last_synced_at`, and `last_error`. No record content is stored. + +**Identity.** The server has a did:web identity derived from `PUBLIC_URL`, for example `did:web:chat.example.com`. It serves `/.well-known/did.json` with one service entry: ID `#scn_chat`, type `AtprotoSpaceService`, endpoint `PUBLIC_URL`. The service identifier is `#scn_chat`. + +**Space credentials.** `registerNotify` only accepts a space credential, even from the space's owner. To get one, the server calls `com.atproto.space.getDelegationToken` on the user's PDS, generates an ES256 key with `JoseKey.generate` from `@atproto/jwk-jose`, and posts to `com.atproto.space.getSpaceCredential` on the user's PDS with a DPoP proof from `createDpopProof` in `@atproto/space`. Credentials last two hours and are cached in memory. A call that fails with an auth error discards the credential and retries once with a fresh one. + +**Watching the index.** Each spaces user's settings space is registered with `com.atproto.space.registerNotify`, at login and then renewed by a job every five minutes when less than an hour remains. That is one registration per user, whatever their number of conversations. Conversations themselves are not registered. + +**Syncing the index.** Syncing a user reads `listOps` on the settings space from its `last_rev`. For each changed `conversationRef`, it syncs that conversation. A deleted `conversationRef` drops the conversation's `sync_state` row. + +A user's first index sync, with no `last_rev`, does not sync every conversation. It sets the cursor at the head of the op log, and only syncs, as backfills, the conversations whose entry's `updatedAt` is inside the backfill window from the chat-turns spec. The rest are synced the first time their entry changes. A returning user with thousands of chats therefore costs a handful of calls at login, not thousands. + +**Coalescing.** Every message write updates the index, and that write notifies the server back. Syncs are therefore coalesced per user: a sync requested while one is running, or within two seconds of the last one, runs once after it. Ops whose CID matches a record the server itself wrote in the last ten minutes, tracked in memory, emit no events. + +**Syncing a conversation.** It reads `listOps` on the conversation from its `last_rev`, advances `last_rev`, and emits a `message:changed` event for each message op. The event carries the conversation, the record, whether it was a create or an update, and whether it is live. It is live when the conversation already had a `last_rev`. It is a backfill when this is the first sync of the conversation, and then the ops cover its whole history. Chat-turns uses that distinction. A record that fails lexicon validation emits a `message:invalid` event instead, with the raw record and the validation error. When an info op carries a title that differs from the `conversationRef`'s copy, the entry is rewritten with the info record's title. + +**Detecting gaps.** `listRepoOps` includes the repo's signed commit when a page reaches the head of the log, and the commit's hash is the set hash over every record's collection, key, and CID. The safety net, and the first sync of a conversation, check it. The server lists the space's records with `excludeValues`, which returns those triples without content, rebuilds the set hash with the `@atproto/space` helpers, and verifies the commit. On a mismatch, it resets that space's cursor. The index cursor moves to the head and the entries inside the backfill window are synced, as on a first sync. A conversation's cursor is cleared, so its next sync is a backfill. Either way a warning is logged. + +**What starts a sync.** + +- A `notifyWrite` for a user's settings space syncs the index. +- `POST /xrpc/network.sharedcomputer.chat.requestSync` syncs the named conversation, or the whole index without one. +- Opening a conversation in the web UI, or pressing its sync button, syncs that conversation. +- The optional safety net, a periodic index sync, as configured below. +- Discovery, as configured below. + +**Development.** A PDS cannot resolve a did:web on a loopback URL, so notifications never arrive in local development. There, sync relies on the safety net, the sync button, and opening a chat. + +**Inbound notifications.** `POST /xrpc/com.atproto.space.notifyWrite` and `POST /xrpc/com.atproto.space.notifySpaceDeleted` verify the service auth token with `verifyJwt` from `@atproto/xrpc-server`. The audience must be the service identifier, the `lxm` must match the method, and the issuer must be the space's authority, with its key resolved through `@atproto/identity`. Valid requests get 200 immediately, and the sync runs in the background. Invalid ones get 401. + +**The sync ping.** `requestSync` authenticates with a service auth token from the caller's PDS, verified with `verifyJwt`. The audience must be the server's DID, the `lxm` must be `network.sharedcomputer.chat.requestSync`, and the issuer must be a spaces account on this server. A named conversation must belong to the issuer, or the call fails with `UnknownConversation`. A conversation the server has never synced is synced from the start, as a backfill. + +**Discovery.** Discovery calls `com.atproto.space.listSpaces` filtered to the conversation type. It adds a `conversationRef` for any conversation that has none, with the title from its info record, and syncs it as a backfill. It runs at every login, and for active users on the configured interval. It repairs the index for clients that skipped the index convention, and for failed index writes. + +**Activity.** `account.last_active_at`, from the auth spec, is updated on any web request, any turn, and any synced change. It decides which users the safety net and discovery cover. + +### Sync configuration + +The admin configures sync in `scn-chat.config.ts`: + +```ts +sync: { + safetyNet: { enabled: true, intervalMinutes: 15, activeWithinHours: 24 }, + discovery: { intervalMinutes: 60, activeWithinDays: 30 }, + allowUserOptOut: true, +} +``` + +- The safety net syncs the index of every user active within `activeWithinHours`, every `intervalMinutes`. That is one PDS call per user per interval, plus one per changed conversation. +- Discovery runs every `intervalMinutes` for users active within `activeWithinDays`. +- When `allowUserOptOut` is true, each user can turn off background sync for their account. The safety net and discovery then skip them, and only notifications, pings, and opening a chat sync their conversations. The setting is the `background_sync` column of `account`. + +### API + +| Route | Purpose | +|---|---| +| `GET /api/conversations?since=` | The chat index, or its changes since a revision | +| `POST /api/conversations` | Create a conversation | +| `GET /api/conversations/:skey?since=` | Info and messages, or their changes since a revision. Also syncs the conversation. | +| `GET /api/conversations/keys`, `GET /api/conversations/:skey/keys` | Every record's collection, key, and CID, without content, from a metadata-only listing. The browser store uses them to reconcile its copy. | +| `PATCH /api/conversations/:skey` | Change title (setting `titleSource: "user"`), tags, or system prompt | +| `DELETE /api/conversations/:skey` | Delete a conversation | +| `POST /api/conversations/:skey/sync` | Sync the conversation now | +| `GET /api/preferences`, `PUT /api/preferences` | Read and write preferences | +| `GET /api/account`, `PUT /api/account` | Read and change account settings, currently `backgroundSync` | +| `GET /api/events` | Server-sent events telling the browser which conversations changed, so it can fetch their changes | + +All routes act on the signed-in user's own data. Message routes are in the chat-turns spec. + +## Scope Boundaries + +- No server-side copy or cache of chat content for spaces users. +- No migrating local users to spaces. +- No syncing repos other than the owner's. +- No notification registrations on individual conversations. +- No multi-instance deployment. Locks, credential caches, and the event bus are in-process. +- No encrypted conversations. + +## Edge Cases and Decisions + +- Reads pass through to the PDS on every request. The browser's own copy, not a server cache, is what makes the UI fast. +- The fallback backend emulates a PDS, including CIDs and an op log, so all code above `RecordStore` is the same for both kinds of user. +- Only the settings space is registered for notifications. Clients that skip the index convention are covered by the sync ping, by opening the chat, by the safety net, and by discovery. +- A conversation's first sync is a backfill. Chat-turns only answers recent messages from a backfill, so history is never replayed. +- The index write after a message write is not atomic with it. Discovery repairs a missing entry. +- Op log gaps are detected with the set hash, computed from metadata-only listings, so no record content is stored or even fetched. + +## Acceptance Criteria + +- [ ] Creating a conversation creates the space, its info record, and its `conversationRef`, for both backends. +- [ ] Writing a message updates the conversation's `conversationRef` with a new `updatedAt`. +- [ ] A title change is copied into the `conversationRef`. +- [ ] Tags are written only to the `conversationRef`, never to the info record. +- [ ] Syncing an info record whose title differs from the entry's copy rewrites the entry. +- [ ] Deleting a conversation deletes the space and its `conversationRef`. +- [ ] `createRecord` throws `RecordExists` for an existing key in both backends. +- [ ] An invalid record is rejected with `InvalidRecord` before any write. +- [ ] `updateInfo` with `unlessUserTitled` skips the write when the stored title source is `user`. +- [ ] The local backend's CIDs match what a PDS computes for the same record. +- [ ] Listing and reading return data from the record store with nothing stored server-side for spaces users. +- [ ] Listing and reading with `since` return only changes after that revision, including deletions. +- [ ] Login creates the settings space when missing and registers it for notifications. +- [ ] `did.json` publishes the `#scn_chat` service entry. +- [ ] A settings-space `notifyWrite` syncs every changed conversation. +- [ ] `notifyWrite` with a bad audience, method, or issuer returns 401. +- [ ] `requestSync` with valid service auth syncs the named conversation, and fails with `UnknownConversation` for someone else's. +- [ ] A conversation's first sync emits backfill events, and later syncs emit live events. +- [ ] An invalid synced message emits `message:invalid` and is not treated as a message. +- [ ] Discovery adds a missing `conversationRef` and syncs the conversation. +- [ ] The safety net and discovery follow the configured intervals, and skip users who turned background sync off. +- [ ] Users cannot turn off background sync when `allowUserOptOut` is false. +- [ ] Registrations are renewed when less than an hour remains. +- [ ] A first index sync only syncs conversations whose entries are inside the backfill window, and sets the cursor at the head. +- [ ] Syncs requested in quick succession for a user run once. +- [ ] Ops matching the server's own recent writes emit no events. +- [ ] A set hash mismatch resets the space's cursor and logs a warning. +- [ ] The keys routes return collections, keys, and CIDs without record content. +- [ ] The owner's writes to the settings space are forwarded to the server under its write policy, checked against a real PDS. + +## Files + +- (to be populated during implementation) diff --git a/specs/chat-turns.md b/specs/chat-turns.md new file mode 100644 index 0000000..7c150cb --- /dev/null +++ b/specs/chat-turns.md @@ -0,0 +1,150 @@ +# Chat Turns + +## Summary + +A turn is one user message and the assistant reply it asks for. The web UI starts a turn by posting a message. A direct-PDS client starts one by writing a user message with a `generation` object, which the server notices through sync. Either way, one function, `startTurn`, runs it. It claims the reply by writing a pending placeholder at the reply's deterministic key, resolves the model, builds the prompt from the conversation's branch, and streams the reply through the Vercel AI SDK. While generating, it forwards progress to the web UI over server-sent events. When the model finishes, it writes the final reply record. Regenerating, editing, and cancelling are variations on the same flow. + +## Motivation + +This is the core of the app. It has to honor the direct-write conventions in `lexicons/README.md` exactly, so a conversation behaves the same whether the web UI or a hand-written record started it. Claiming the reply with a create-only write at a deterministic key guarantees one generation per attempt, even when the web UI and sync notice the same message. + +## Design + +### Starting a turn from the web UI + +`POST /api/conversations/:skey/messages` with: + +```ts +{ parent?: string; parts: Part[]; generation?: { model?: ModelRef; effort?: Effort; tools?: string[] } } +``` + +`parts` are text parts plus the attachment parts returned by the attachments API. The server builds the user message record, with a new TID key, `role: "user"`, the parent, `content` as `plainContent`, the generation request with `attempt` 0, and `createdAt`. It writes the record with `createMessage`, then calls `startTurn` when a generation request is present. The response returns the user message key and the reply key. + +An edit is a new message with the same parent as the one being edited. The web UI sends it through this same route. + +`POST /api/conversations/:skey/messages/:rkey/regenerate` with an optional `model` and `effort` rewrites the user message with `putMessage`. It raises `generation.attempt` by one and applies the new model and effort. Then it calls `startTurn`. + +`POST /api/conversations/:skey/messages/:rkey/cancel` aborts the running generation for that reply key, if this server is running it. + +### Starting a turn from a direct write + +Chat-turns listens for `message:changed` events from sync. It calls `startTurn` when all of these hold: + +- The message is a create or an update of a `user` message authored by the conversation's owner. +- The message has a `generation` object. +- No message exists at the reply key for its current attempt. +- The event is live, or the message's `createdAt` is within the last `TURN_BACKFILL_MINUTES`, default 60. + +A conversation's first sync is a backfill, which only starts turns for recent messages. A conversation created by another client still gets answered, while importing or migrating old messages never triggers a flood of generations. + +Whether a reply exists is checked with `getRecord` at the reply key, through the record store. + +A turn that has been requested but not yet claimed, for example one waiting for the rate limit, is recorded in a `turn_request` table: conversation URI, user message key, attempt, and time, with the first three as the primary key. The same turn requested twice, say by the safety net and a notification, is therefore queued once. The row is deleted when the reply is claimed. On startup, the server retries every row newer than `TURN_BACKFILL_MINUTES` and drops older ones, so a restart does not lose queued turns. The table holds references only, never content. + +When sync emits `message:invalid` for a user message that carries a generation request, the server writes an error reply at `.r`, using the attempt from the raw record or 0. Its error message quotes the validation error, so the direct client sees why nothing was generated. + +### startTurn + +`startTurn(user, conversation, userMessageKey)` runs the turn in the background and returns once the reply is claimed: + +1. **Load.** It reads the conversation's info record and every message from the record store, once per turn, and keeps them in memory only for the turn. The reply key is `.r`, with `attempt` defaulting to 0. +2. **Choose the model.** The model reference is `generation.model`, else the model of the nearest completed assistant reply up the branch, else the user's `defaultModel` preference, else the admin default. The effort is `generation.effort`, else the `defaultEffort` preference. +3. **Rate limit.** A user may start `TURN_RATE_PER_MINUTE` turns per minute, default 10. Past that, the turn waits in an in-memory queue per user and starts when the limit allows, without claiming the reply yet. The web UI's stream shows a `queued` status meanwhile. +4. **Claim.** It calls `createMessage` at the reply key with a placeholder: `role: "assistant"`, the parent key, empty `plainContent` parts, the chosen model and effort, `status: "pending"`, and `createdAt`. If that throws `RecordExists`, another runner, possibly another deployment watching the same space, owns the attempt, and `startTurn` stops. On success it records the reply URI in a local `turn_claim` table. +5. **Resolve.** The providers spec resolves the model reference to an AI SDK model. A failure rewrites the placeholder with `status: "error"` and a message saying why. +6. **Build the prompt.** The branch is the chain of parents from the user message to the root. The instructions are the `customInstructions` preference and the conversation's `systemPrompt`, joined by a blank line. Each message becomes an AI SDK message: + - User text parts become text parts. Attachments follow the attachments spec. + - Assistant replies with status `complete` or `cancelled` contribute their text parts, tool calls and results, and reasoning according to the provider's replay policy. Replies with status `error` or `pending` are skipped. +7. **Filter.** It runs the `messages:beforeModel` hook. +8. **Generate.** It calls `streamText` with the model, instructions, messages, the requested tools that are registered, `stopWhen: isStepCount(TURN_MAX_STEPS)` (default 8), the mapped `reasoning` value, and an abort signal. The signal fires on cancel or after `TURN_TIMEOUT_SECONDS`, default 600. +9. **Stream.** It reads `result.stream`, builds the reply's parts, and publishes events to the turn's stream buffer. +10. **Finish.** It builds the final record: parts in the order the model produced them, usage from `result.usage` (input, output, and `outputTokenDetails.reasoningTokens`), and a status. It runs the `message:afterModel` hook, then writes the record with `putMessage`. After that it runs the `turn:after` hook. + +### Mapping stream parts to lexicon parts + +| AI SDK stream parts | Lexicon part | +|---|---| +| `text-delta` runs between `text-start` and `text-end` | `textPart` | +| `reasoning-delta` runs between `reasoning-start` and `reasoning-end` | `reasoningPart` | +| `tool-call` | `toolCallPart`, with `input` JSON-encoded | +| `tool-result` or `tool-error` | `toolResultPart`, with `output` JSON-encoded unless already a string, and `isError` for errors | +| `source` with `sourceType: "url"` | `sourcePart` | + +Other parts are ignored. When a stream part carries `providerMetadata`, it is JSON-encoded into the lexicon part's `providerData`. For text and reasoning, the metadata from the part's end event wins, since that is where providers attach signatures. + +### Ending states + +- **Complete.** The stream finished normally. Status `complete`. +- **Error.** The stream yielded an `error` part, or anything in the turn threw. Status `error`, with a short message from the error that never includes API keys or request headers. The partial parts are kept. +- **Cancelled.** The abort signal fired. Status `cancelled`, with the partial parts kept. +- **Too large.** If the final record's JSON is over 900 KB, the largest tool result outputs are uploaded as blobs and referenced from each part's `outputBlob`, and `output` keeps a short summary. If the record is still too large, the turn ends with status `error`. + +On startup, the server rewrites any `pending` reply in its own `turn_claim` table as `error` with the message "interrupted", then clears the table. Pending replies claimed by other deployments are never touched. + +### Streaming to the web UI + +`GET /api/conversations/:skey/messages/:rkey/stream` is a server-sent event stream for a reply key, built with Hono's `streamSSE`. It first replays the events buffered so far, then sends live events: + +| Event | Data | +|---|---| +| `part-start` | The part index and type | +| `delta` | The part index and appended text | +| `part` | A finished tool call, tool result, or source part | +| `queued` | The turn is waiting for the rate limit | +| `status` | The final status and error, if any | + +The stream closes after `status`. If this server is not running that reply, the stream sends one `status` event with the reply's stored status and closes. The web UI then reads the record through the conversation API. A buffer is dropped a minute after its turn ends. + +### Turn context + +Hooks receive a turn context holding the user's DID, the conversation's URI and info, the user's preferences, the user message, the reply record as it stands, the resolved model reference and provider, and the effort. + +## Scope Boundaries + +- No multi-instance deployment. Streams, cancellation, and rate limits are in-process. +- No built-in tools. Tools come from plugins. +- No tool approval prompts. +- No streaming for direct-PDS clients. They see the placeholder, then the finished record. +- No context window management beyond what the provider enforces. + +## Edge Cases and Decisions + +- The pending placeholder is written with a create-only call, so the web UI and sync noticing the same message produce one generation. +- Rebuilds and backfills only start turns for messages newer than the backfill window, so missed notifications are recovered without replaying history. +- A rate-limited turn is delayed, not answered with an error, so direct clients are not left with a dead-end reply. +- Several deployments can watch the same spaces. The create-only claim picks one winner, and each server only cleans up its own claims. +- Replies that errored or are still pending are left out of later prompts. Cancelled replies keep their partial content in the prompt. +- The model falls back to the last completed reply's model on the branch, so switching models sticks for later turns, as the lexicons README says. +- A turn interrupted by a restart becomes `error` with "interrupted" on the next startup, not a retry, because retrying could spend money twice. + +## Acceptance Criteria + +- [ ] Posting a message with a generation request writes the user message, a pending reply at `.r0`, and then the final reply. +- [ ] Posting without a generation request writes only the user message. +- [ ] A direct-write user message with a generation request, seen through live sync, starts a turn. +- [ ] An old unanswered user message seen during a backfill does not start a turn. +- [ ] A message authored by someone other than the owner does not start a turn. +- [ ] When the web UI and sync both notice a message, exactly one generation runs. +- [ ] Regenerating raises the attempt and writes the reply at `.r`, keeping earlier replies. +- [ ] An edit creates a sibling message with the same parent and its own reply. +- [ ] Model resolution follows request, then last completed reply on the branch, then preferences, then the admin default. +- [ ] The prompt contains only the branch from the root to the user message, with custom instructions and the system prompt as instructions. +- [ ] Error and pending replies are left out of the prompt. +- [ ] Stream parts map to lexicon parts as in the table, with tool input and output JSON-encoded. +- [ ] A provider error ends the turn with status `error`, a message without secrets, and the partial parts. +- [ ] Cancelling ends the turn with status `cancelled` and the partial parts. +- [ ] A turn past the timeout is cancelled. +- [ ] A final record over 900 KB moves the largest tool outputs into `outputBlob` blobs. +- [ ] The stream endpoint replays buffered events to a late subscriber, then sends live ones. +- [ ] Past the rate limit, the turn waits unclaimed and starts once the limit allows. +- [ ] A backfill starts turns for unanswered messages newer than the backfill window, and not for older ones. +- [ ] On startup, requested but unclaimed turns newer than the backfill window are retried, and older ones are dropped. +- [ ] The same unclaimed turn requested twice is queued once. +- [ ] A turn reads the conversation from the record store and stores no content server-side. +- [ ] An invalid direct-write message with a generation request gets an error reply quoting the validation error. +- [ ] On startup, pending replies this server claimed are marked `error` with "interrupted", and pending replies it did not claim are left alone. +- [ ] The `turn:after` hook runs after the final record is written, with the turn context. + +## Files + +- (to be populated during implementation) diff --git a/specs/foundation.md b/specs/foundation.md new file mode 100644 index 0000000..54fd393 --- /dev/null +++ b/specs/foundation.md @@ -0,0 +1,151 @@ +# Foundation + +## Summary + +The foundation is the empty but working skeleton every later spec builds on. It sets up a pnpm workspace with three packages: generated TypeScript for our lexicons, a Hono server, and a Vite React web app. It adds environment config, a Kysely database layer that runs on SQLite or Postgres, a migration runner, and a test setup that exercises both databases. When it is done, `pnpm dev` starts the server and web app, the web page shows the server's health, and `pnpm test` passes on both databases. No chat features exist yet. + +## Motivation + +Every phase 1 spec needs the same plumbing: config, a database, a server, a web app, and tests. Settling it once keeps later specs about features rather than setup. The foundation also locks in the rules that are painful to retrofit: all lexicon NSIDs come from one generated module, all branding comes from config, the database code stays portable across SQLite and Postgres, and the atproto alpha packages install reproducibly. + +## Design + +### Workspace layout + +``` +package.json root scripts, pnpm overrides, packageManager, engines +pnpm-workspace.yaml packages/*, apps/*, plugins/* +tsconfig.base.json shared strict compiler options +vitest.config.ts one Vitest run across all packages +.nvmrc 24 +.env.example every variable, with safe development defaults +lexicons/ lexicon JSON (exists today) +packages/lexicons/ @scn-chat/lexicons: generated schemas and types +apps/server/ @scn-chat/server: Hono API server +apps/web/ @scn-chat/web: Vite React app +``` + +All packages are ESM (`"type": "module"`) with TypeScript `strict` on. Workspace packages export their TypeScript source directly (`"exports": "./src/index.ts"`), so nothing needs a build step. + +The server runs TypeScript natively with Node's built-in type stripping, in development and production, with no compile step and no runner like tsx. That requires Node 22.18 or newer, and the project targets Node 24, the current LTS. The shared tsconfig sets `erasableSyntaxOnly`, `verbatimModuleSyntax`, and `allowImportingTsExtensions`, so the compiler rejects syntax Node cannot strip, such as enums and namespaces. Relative imports use `.ts` extensions. `tsc --noEmit` does type checking. Vite handles TypeScript for the web app. + +The plugin API package and the `plugins/` directory are created by the plugins spec. + +### atproto alpha packages + +Every `@atproto/*` package with an alpha release is pinned to one exact alpha version through `pnpm.overrides` in the root `package.json`. A range like `^0.0.0-spaces-alpha-*` also matches the broken `0.0.0` releases, so ranges are never used. The lockfile is committed. Upgrading the alpha means changing one version string for every override at once. + +### Lexicon package + +`packages/lexicons` holds TypeScript generated from `lexicons/` by the `lex build` command from `@atproto/lex`, the same tool bulletin uses. Its schemas come from `@atproto/lex-schema`. The package's index is the one module the rest of the code uses for NSIDs, record types, and record validation. No other code writes an NSID as a string. + +- `pnpm codegen` regenerates it with `--clear`, `--index-file`, and `--import-ext .ts`. +- The generated files are committed, so reviewers see schema changes in diffs and nothing needs generating after a clone. +- Upstream lexicons that later specs need, such as `com.atproto.space.*`, are vendored into `lexicons/upstream/` when that spec needs them, with a note recording the source commit, as bulletin does. + +### Config + +`apps/server/src/config.ts` parses `process.env` with a zod schema once at startup and exports a typed, frozen config object. An invalid or missing required value stops the server with a message naming the variable. `.env` is loaded with Node's built-in `process.loadEnvFile` when the file exists. + +| Variable | Default in `.env.example` | Purpose | +|---|---|---| +| `NODE_ENV` | `development` | `development`, `test`, or `production` | +| `PORT` | `3000` | Server port | +| `PUBLIC_URL` | `http://127.0.0.1:3000` | Public base URL of the app | +| `APP_NAME` | `SCN Chat` | Display name, used by the web app | +| `DATA_DIR` | `./data` | Directory for local files such as the SQLite database and blobs | +| `DATABASE_URL` | `sqlite:./data/scn-chat.sqlite` | `sqlite:` or `postgres://...` | +| `LOG_LEVEL` | `info` | pino log level | + +Later specs add their own variables to the same schema and to `.env.example`. + +### Database + +`apps/server/src/db/` creates a Kysely instance from `DATABASE_URL`: + +- `sqlite:` uses `better-sqlite3` with `SqliteDialect`, with WAL mode and foreign keys turned on. +- `postgres://` uses `pg` with `PostgresDialect`. + +The database types live in one `Database` interface that later specs extend. + +Migrations are TypeScript files in `apps/server/src/db/migrations/`, registered in an explicit ordered map and run by Kysely's `Migrator`. The server runs pending migrations on startup, and `pnpm migrate` runs them on their own. The foundation ships the runner with no tables. Each later spec adds its own migrations. + +Migrations and queries stay within what both databases support. Timestamps are ISO 8601 text. JSON, including stored lexicon records, is text. Any dialect-specific SQL has to live behind a helper with a test on both databases. + +### Server + +`apps/server` uses `hono` with `@hono/node-server`. `createApp(deps)` builds the Hono app from a config and a database, so tests can create an app without opening a port. The entry point creates the config, database, and logger, runs migrations, and starts listening. + +- `GET /api/health` returns `{ "status": "ok", "appName": "..." }` after a trivial database query succeeds. It returns 503 with `{ "status": "error" }` when the query fails, and logs the error. +- Every route under `/api` is private to our web app. None of it is a public API. +- In production the server also serves the built web app from `apps/web/dist`, with an `index.html` fallback for client-side routes. In development Vite serves the web app and proxies `/api` to the server. +- Logging uses pino, with request logging through a small Hono middleware. + +### Web app + +`apps/web` is a Vite React app with no router, state library, or CSS framework. The single `App` component fetches `/api/health` and shows the app name and whether the server is reachable. There is no styling beyond browser defaults. + +### Tests + +Vitest runs every package from the root config. Tests live in a `tests/` directory in each package that mirrors its `src/`. + +- Database tests run against both SQLite and Postgres through a shared helper. SQLite uses an in-memory `better-sqlite3` database. Postgres uses PGlite, an in-process build of Postgres, through `kysely-pglite-dialect`, so no Docker or database server is needed. +- Web tests use jsdom and Testing Library. +- The lexicon package has tests that every lexicon document validates against the alpha lexicon schema, and that the generated code is current with the JSON. + +### Scripts + +| Root script | Does | +|---|---| +| `pnpm dev` | Runs the server with `node --watch` and the Vite dev server together | +| `pnpm build` | Builds the web app | +| `pnpm start` | Runs the server in production mode, serving the built web app | +| `pnpm test` | Runs all tests once | +| `pnpm typecheck` | Type checks every package | +| `pnpm lint` | Lints and checks formatting with Biome | +| `pnpm format` | Formats with Biome | +| `pnpm check` | Runs lint, typecheck, and test, for any CI system | +| `pnpm codegen` | Regenerates the lexicon package | +| `pnpm migrate` | Runs pending migrations | + +### Linting and continuous integration + +Biome handles both linting and formatting, configured in one `biome.json` at the root. No CI configuration is committed, so the project is not tied to one host. Any CI system runs `pnpm install --frozen-lockfile` and then `pnpm check`. + +## Scope Boundaries + +- No authentication, sessions, or OAuth. That is the auth spec. +- No database tables. Each later spec adds its own. +- No plugin interfaces or plugin package. That is the plugins spec. +- No atproto network calls and no vendored upstream lexicons yet. +- No Docker image or deployment setup. +- No CI configuration for any particular host. +- No UI beyond the health status page. + +## Edge Cases and Decisions + +- The server runs TypeScript through Node's native type stripping instead of compiling to JavaScript. If the generated lexicon code turns out to use syntax Node cannot strip, the fallback is running the server through `tsx`. +- Node refuses to strip types for files under `node_modules`. Workspace packages work because pnpm symlinks them there and Node resolves symlinks to their real paths by default. Never run with `--preserve-symlinks`. +- SQLite uses `better-sqlite3`, not Node's built-in `node:sqlite`, because Kysely's SQLite dialect targets the `better-sqlite3` API. It ships prebuilt binaries for common platforms. +- Generated lexicon code is committed, with a test that fails when it is stale. +- Migrations run on server startup. Kysely's migrator takes a lock, so several instances starting together on Postgres do not run a migration twice. +- Timestamps and JSON are stored as text on both databases, trading Postgres-native types for portability. Search and other features that need native types add them behind dialect helpers. + +## Acceptance Criteria + +- [ ] `pnpm install` on a clean clone succeeds with the committed lockfile, and resolves no `@atproto/*` package to a plain `0.0.0` release. +- [ ] Config parsing returns typed values for a valid environment, and fails with the variable's name when a required value is missing or invalid. +- [ ] `DATABASE_URL` with `sqlite:` produces a working SQLite database, and with `postgres://` produces a working Postgres database. +- [ ] An unsupported `DATABASE_URL` scheme fails at startup with a clear message. +- [ ] The migrator applies pending migrations in order on both databases, and running it again applies nothing. +- [ ] `GET /api/health` returns 200 with the app name when the database is reachable. +- [ ] `GET /api/health` returns 503 when the database query fails. +- [ ] In production mode, the server serves the built web app and falls back to `index.html` for unknown non-API paths. Unknown `/api` paths return 404. +- [ ] The web app renders the app name and the server's health status, and shows an unreachable state when the health request fails. +- [ ] Every lexicon document validates against the alpha lexicon schema. +- [ ] The generated lexicon code matches a fresh `pnpm codegen` run. +- [ ] `pnpm check` passes on a clean clone. + +## Files + +- (to be populated during implementation) diff --git a/specs/plugins.md b/specs/plugins.md new file mode 100644 index 0000000..45b6664 --- /dev/null +++ b/specs/plugins.md @@ -0,0 +1,160 @@ +# Plugins + +## Summary + +SCN Chat's features beyond the core chat loop come from plugins: model providers, tools the model can call, file ingesters that turn uploads into text, and hooks that run around each turn. A plugin is an npm package that exports a factory. The admin lists plugins in a TypeScript config file, `scn-chat.config.ts`, much like `vite.config.ts`. At startup the server imports the config, checks each plugin, and calls its `setup` function, which registers what the plugin provides. The types plugins build against live in their own package, `@scn-chat/plugin-api`, so third parties can write plugins without depending on the server. + +## Motivation + +The project is meant to be extended and forked. Anyone should be able to add a model provider, a search engine, or an OCR engine without touching core code. Other chat apps either have no code-level plugin system (Chatbot UI, LibreChat, which relies on YAML config and MCP) or store pasted code in the database without versioning (Open WebUI). Using npm packages and a typed config file gives versioning, lockfiles, type-checked options, and ordinary code review for free. + +The design borrows from several systems: + +- **Drupal's split between registries and hooks.** Some extension points pick one implementation by ID (Drupal's Plugin API). Others let many plugins act on the same event (Drupal hooks, WordPress actions and filters). +- **Vite's plugin shape.** A plugin is a factory returning a plain typed object, and the config lists plugins in order. +- **Open WebUI's settings split.** Admin settings and per-user settings are separate, and forms are generated from a schema. +- **Fastify and VS Code's compatibility checks.** Each plugin declares which plugin API version it targets, and incompatible plugins are refused at startup. + +## Design + +### Packages + +- `packages/plugin-api` (`@scn-chat/plugin-api`) exports `definePlugin`, `defineConfig`, and every type a plugin sees. It depends only on `zod`, `@scn-chat/lexicons`, and `@ai-sdk/provider` for the model interface. It has its own semver version, separate from the app. Inside the workspace it exports TypeScript source like every other package. Publishing it to npm for third-party authors, built to JavaScript with type declarations, is a later step. +- `plugins/*` holds the plugins that ship with SCN Chat, each its own workspace package named `@scn-chat/plugin-`. The specs for providers, attachments, and titles define the phase 1 plugins. +- The server depends on `@scn-chat/plugin-api` and implements the registries, hooks, and context behind it. + +### Config file + +`scn-chat.config.ts` at the repository root. The `SCN_CHAT_CONFIG` environment variable can point at another path. A committed `scn-chat.config.example.ts` shows a working setup, and the real file is gitignored. The server fails at startup with a message pointing at the example when the config file is missing. + +```ts +import { defineConfig } from '@scn-chat/plugin-api' +import anthropic from '@scn-chat/plugin-anthropic' +import openaiCompatible from '@scn-chat/plugin-openai-compatible' +import pdfText from '@scn-chat/plugin-pdf-text' +import titles from '@scn-chat/plugin-titles' + +export default defineConfig({ + plugins: [ + anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }), + openaiCompatible({ id: 'cocore', name: 'co/core', baseURL: 'https://cocore.dev/v1', apiKey: process.env.COCORE_API_KEY }), + pdfText(), + titles(), + ], + models: [ + { provider: 'anthropic', id: 'claude-sonnet-5', name: 'Claude Sonnet 5', capabilities: { vision: true, reasoning: true, tools: true }, roles: ['user'] }, + ], + roles: { + staff: ['did:plc:abc...'], + }, +}) +``` + +Options passed to a plugin factory are the plugin's admin settings. They are typed by the factory's signature, and secrets come from environment variables, never from the file itself. The `models` list is described in the providers spec, `roles` in the auth spec, and `sync` in the chat-storage spec. + +### Plugin shape + +```ts +export default function titles(options: TitlesOptions = {}) { + return definePlugin({ + id: 'titles', + name: 'Chat titles', + apiVersion: 1, + setup(ctx) { + ctx.hooks.on('turn:after', async (turn) => { /* ... */ }) + }, + }) +} +``` + +- `id` is unique across loaded plugins, and matches `^[a-z0-9-]+$`. +- `apiVersion` is the plugin API major version the plugin targets. The server supports exactly version 1 in phase 1. +- `userSettings` is an optional zod schema for per-user settings. Fields marked with `.meta({ secret: true })` are stored encrypted. +- `setup(ctx)` runs once at startup and may be async. Startup waits for every `setup` to finish, in config order. + +### Plugin context + +`setup` receives a narrow context. It has no raw database handle and no access to OAuth sessions, so a plugin cannot bypass record ownership by accident. + +| Member | Purpose | +|---|---| +| `ctx.providers.register(provider)` | Add a model provider. See the providers spec. | +| `ctx.tools.register(tool)` | Add a tool the model can call. | +| `ctx.toolSources.register(source)` | Add a source of tools that change at runtime, such as an MCP server. | +| `ctx.ingesters.register(ingester)` | Add a file ingester. See the attachments spec. | +| `ctx.hooks.on(name, handler, options?)` | Handle a hook. | +| `ctx.models.generateText(request)` | Run a one-off model call for a user, through the provider registry. | +| `ctx.conversations.updateInfo(user, conversation, patch)` | Change a conversation's info record, written under the user's DID through the storage layer. | +| `ctx.userSettings(user)` | Read this plugin's validated settings for a user. | +| `ctx.logger` | A pino child logger tagged with the plugin ID. | +| `ctx.app` | Read-only app name and public URL. | +| `ctx.onClose(fn)` | Register cleanup to run at shutdown. | + +### Registries + +Registries hold implementations picked by ID or by match. Registering a duplicate ID fails startup. + +- **Providers**, picked by provider ID. The providers spec defines the interface. +- **Tools**, picked by name. A tool has a `name`, a `description`, a zod `inputSchema`, and `run(input, context)`, which returns text or JSON-encodable data. A result that would push the record past the size cap is stored as a blob. +- **Tool sources**, each with `list(user)` and `call(user, name, input)`. Their tools are merged with registered tools, with names prefixed by the source ID. +- **Ingesters**, matched by MIME type. The attachments spec defines the interface. + +In phase 1 no tools are registered. The interfaces exist so web search, fetch, and MCP plugins can be added without core changes. + +### Hooks + +Hooks let many plugins act on the same event. Every payload uses our own normalized types, never a provider's raw request or response shape. + +| Hook | Kind | Payload | +|---|---|---| +| `messages:beforeModel` | filter | The system prompt and message list about to be sent, with the turn context | +| `message:afterModel` | filter | The assistant's final content before it is written, with the turn context | +| `turn:after` | action | The finished turn: conversation, user, request, and the written reply | +| `conversation:created` | action | The new conversation and its owner | +| `conversation:deleted` | action | The deleted conversation and its owner | + +- **Filters** run in order, and each handler returns the value the next one receives. A filter that throws fails the turn, which ends with status `error` and an error naming the plugin. +- **Actions** run in order, and their return values are ignored. An action that throws is logged with the plugin ID, and the turn is unaffected. +- **Order.** Handlers are sorted by `options.order` (`'pre'`, `'normal'`, or `'post'`, default `'normal'`), then by the plugin's position in the config. There are no numeric priorities. The server logs each hook's resolved handler order at startup. + +The hook runner is a small typed module in the server, about 80 lines, that handles ordering, filter chaining, and action error isolation. The `hookable` library has no ordering and cannot pass a value through a chain of handlers, which filters need. `tapable` has both, but needs a class per hook. + +### Per-user settings + +Per-user settings are stored in a `plugin_user_settings` table: DID, plugin ID, JSON value, and a separate encrypted JSON value for secret fields. Values are validated against the plugin's schema on read. Invalid stored values fail loudly and are not silently reset. The settings page shows the validation error for that plugin with a reset button, which calls `DELETE /api/plugins/:id/settings` and clears the user's stored values for it. The web UI renders a settings form from `z.toJSONSchema(userSettings)`, as the web-ui spec describes. Secret fields are encrypted with the same key and scheme as BYO API keys, described in the providers spec. + +## Scope Boundaries + +- No admin UI for installing or configuring plugins. Admins edit the config file and restart. +- No sandboxing. Plugins run in-process with full trust, and the docs say so. +- No user-installed plugins. If users ever add their own tools, that goes through MCP, not in-process code. +- No plugins that add UI to the web app. +- No filter over individual streaming chunks in phase 1. +- No built-in tools in phase 1. + +## Edge Cases and Decisions + +- Plugins are listed explicitly in the config file, not discovered by scanning `node_modules`, which is unreliable under pnpm. +- Hook ordering uses config position plus pre, normal, and post, not numeric priorities, to avoid WordPress's priority guessing. +- Filter errors fail the turn, and action errors do not. A filter changes what the model sees or what gets written, so skipping one silently could corrupt a conversation. +- Plugin output that becomes a record goes through the context's storage methods, so it is always written under the user's DID. + +## Acceptance Criteria + +- [ ] The server loads plugins from the config file in order and calls each `setup` once. +- [ ] A missing config file fails startup with a message naming the example file. +- [ ] Two plugins with the same ID fail startup, naming the ID. +- [ ] A plugin with an unsupported `apiVersion` fails startup, naming the plugin and version. +- [ ] Registering a duplicate provider ID, tool name, or ingester ID fails startup. +- [ ] Filter handlers run in order, with pre before normal before post, and config order within each group. +- [ ] Each filter receives the previous filter's return value. +- [ ] A throwing filter fails the turn with an error naming the plugin. +- [ ] A throwing action is logged and does not affect the turn or later actions. +- [ ] Per-user settings round-trip through storage, with secret fields stored encrypted and returned decrypted. +- [ ] Stored per-user settings that fail validation raise an error instead of being reset. +- [ ] Deleting a plugin's settings clears the user's stored values, so the schema defaults apply. +- [ ] `ctx.conversations.updateInfo` writes through the storage layer under the user's DID. + +## Files + +- (to be populated during implementation) diff --git a/specs/providers.md b/specs/providers.md new file mode 100644 index 0000000..fb5a978 --- /dev/null +++ b/specs/providers.md @@ -0,0 +1,160 @@ +# Providers + +## Summary + +A provider connects SCN Chat to a model API. Providers are plugins that register with the provider registry from the plugins spec. Each provider turns a model ID and an API key into a Vercel AI SDK language model, and the chat-turns spec drives that model. Admins configure providers and a list of models everyone can use in `scn-chat.config.ts`. Users can add their own API keys for any provider that allows it, and, when the admin permits, their own OpenAI-compatible endpoints. User keys are encrypted in the app database. Phase 1 ships four provider plugins: Anthropic, OpenAI, Google, and OpenAI-compatible, which covers OpenRouter, co/core, llama.cpp, vLLM, and Ollama. + +## Motivation + +Users should be able to chat with any model they have access to, and switch models mid-conversation. Admins of an instance should be able to offer models to everyone. The AI SDK already handles streaming, tool calls, reasoning, and provider differences for dozens of APIs, so each provider plugin is a thin wrapper around one AI SDK provider package. + +## Design + +### Provider interface + +In `@scn-chat/plugin-api`: + +```ts +interface ModelProvider { + id: string // matches modelRef.provider, e.g. 'anthropic', 'cocore' + name: string // shown in the UI + userKeys: boolean // whether users may add their own API key + userEndpoints?: boolean // whether users may add their own base URLs (OpenAI-compatible only) + createModel(args: { modelId: string; apiKey?: string; baseURL?: string }): LanguageModelV4 + listModels?(args: { apiKey?: string; baseURL?: string }): Promise + replay: ReplayPolicy // how earlier reasoning is sent back, see below +} + +interface ModelInfo { + id: string + name: string + capabilities: { vision: boolean; reasoning: boolean; tools: boolean } +} +``` + +`LanguageModelV4` is the AI SDK's provider specification, from `@ai-sdk/provider`. That package is separate from the `ai` runtime and versions the interface explicitly. Returning a model that implements it, instead of our own streaming interface, means any AI SDK provider package, including community ones, can become an SCN Chat provider in a few lines. The rest of our interface (IDs, keys, listing, replay) stays ours. + +`createModel` is called per request, since API keys can differ per user. The AI SDK factories are cheap to call. + +### Phase 1 provider plugins + +| Package | AI SDK package | Notes | +|---|---|---| +| `@scn-chat/plugin-anthropic` | `@ai-sdk/anthropic` | `createAnthropic({ apiKey })` | +| `@scn-chat/plugin-openai` | `@ai-sdk/openai` | Responses API, with `store: false` and `include: ['reasoning.encrypted_content']` so reasoning can be replayed without OpenAI storing it | +| `@scn-chat/plugin-google` | `@ai-sdk/google` | `createGoogle({ apiKey })` | +| `@scn-chat/plugin-openai-compatible` | `@ai-sdk/openai-compatible` | Takes `id`, `name`, `baseURL`, and optional `apiKey`. Can be listed several times with different IDs. Sets `includeUsage: true` | + +Each factory takes an optional admin `apiKey` and `userKeys` (default `true`). The OpenAI-compatible factory also takes `userEndpoints` (default `false`) and `allowPrivateNetworks` (default `false`). + +### Models + +The config's `models` list names the models every signed-in user can use with the admin's keys: + +```ts +models: [ + { provider: 'anthropic', id: 'claude-sonnet-5', name: 'Claude Sonnet 5', + capabilities: { vision: true, reasoning: true, tools: true }, roles: ['user'], default: true }, +] +``` + +- Startup fails if a model names a provider that is not registered, or a provider without an admin key. +- Exactly zero or one model has `default: true`. It is the fallback when a request names no model, the conversation has no earlier reply, and the user's preferences set no default. + +Each admin model's `roles` lists the roles allowed to use it, using the roles from the auth spec. `roles: ['user']` opens a model to every signed-in user. `roles` is required, so who can use a model is always a deliberate choice, and a model without it fails startup. A model naming an undefined role also fails startup. Roles never restrict a user's own keys. + +### User credentials + +A user adds a credential for a provider that allows user keys. It holds an API key, an optional display name, a base URL when the provider allows user endpoints, and the model IDs the user wants to use, with their capabilities. When the provider implements `listModels`, the UI offers its list. + +Credentials are stored in a `provider_credential` table: ID, owner DID, provider ID, name, slug, base URL, encrypted API key, models as JSON, and timestamps. A user has at most one credential per provider ID, except on providers with user endpoints, where each credential is its own endpoint. The user picks a slug for each endpoint, unique among their credentials and matching `^[a-z0-9-]+$`, and the model reference's provider value becomes `user:`. Records therefore name something the user chose, not a database ID, which survives deleting and re-adding the endpoint. + +### Secret encryption + +`apps/server/src/secrets.ts` encrypts API keys and secret plugin settings with AES-256-GCM from `node:crypto`. The key is a 32-byte value in the `SECRET_KEY` environment variable, base64 encoded, and startup fails without it. Each value is stored as `v1::::`, where the key ID is the first eight hex characters of the key's SHA-256. Values encrypted with a retired key stay readable while that key is listed in `SECRET_KEYS_OLD`, a comma-separated list, so the key can be rotated later without a migration. The version prefix lets the scheme itself change later. Keys are never logged, never returned by the API, and shown in the UI only as their last four characters. + +### Private network guard + +Requests to a user-supplied base URL go through a dedicated `fetch` passed to the AI SDK provider factory. It refuses loopback, private, link-local, and unique-local addresses, unless the provider was configured with `allowPrivateNetworks: true`, in three places. The parsed URL's host is checked first when it is an IP literal, including IPv4-mapped IPv6 addresses, since no DNS lookup happens for those. The undici `Agent`'s DNS lookup checks every resolved address. Its connect step checks the socket's actual remote address before sending anything. The last two mean the address that was checked is the one connected to, so DNS rebinding cannot slip past. Redirects are not followed. That stops users from making the server fetch internal services. Admin-configured base URLs are trusted. + +### Resolving a model reference + +Given a user and a `modelRef` `{ provider, id }`: + +1. If the user has a credential for `provider` that lists `id`, use it. +2. Otherwise, if `{ provider, id }` is in the admin `models` list and one of the user's roles is allowed to use it, use the admin key. +3. Otherwise fail with `ModelUnavailable`, naming the model. + +The user's own key wins over the admin's, so users who bring a key pay for their own usage. + +### Effort + +The lexicon's effort maps onto the AI SDK's cross-provider `reasoning` option. Each AI SDK provider package translates that into its own setting, such as Anthropic's thinking effort, OpenAI's reasoning effort, or Google's thinking level. + +| Lexicon | AI SDK `reasoning` | +|---|---| +| absent | `'provider-default'` | +| `none` | `'none'` | +| `low` | `'low'` | +| `medium` | `'medium'` | +| `high` | `'high'` | +| `max` | `'xhigh'` | + +An unknown value maps to `'provider-default'` with a logged warning. Effort is not sent to models without the reasoning capability. + +### Replaying earlier reasoning + +Some providers require their earlier reasoning to be sent back on later turns, carrying provider data exactly as it was returned. The AI SDK exposes that data as `providerMetadata` on streamed parts, and takes it back as `providerOptions` on message parts. + +Each provider declares a `ReplayPolicy`: + +- **Anthropic:** replay reasoning parts with their signature. Redacted reasoning has empty text and replays its encrypted data. +- **OpenAI:** replay reasoning with its item ID and encrypted content. +- **Google:** replay thought signatures, which Google attaches to text and tool-call parts as well as reasoning. +- **OpenAI-compatible:** drop earlier reasoning. Most open models expect it removed from history. + +Reasoning from a different provider than the current turn's is always dropped. + +The provider data is stored in the `providerData` field of reasoning, text, and tool-call parts, as the JSON-encoded `providerMetadata` the AI SDK returned for that part. On replay, it is decoded and passed back as the part's `providerOptions`. Provider data from a different provider than the current turn's is never sent. + +## Scope Boundaries + +- No admin UI. Providers and admin models are configured in the config file. +- No usage limits, quotas, or billing for admin models beyond role checks. +- No automatic model discovery for admin models. They are listed explicitly. +- No image generation, speech, or embedding models. + +## Edge Cases and Decisions + +- A user's own key takes priority over the admin key for the same model. +- Private network addresses are refused for user endpoints by default, and allowed per provider for self-hosters running local models. +- Provider data is kept as an opaque JSON string per part, so any provider's replay data fits without lexicon changes. +- The plugin API depends on `@ai-sdk/provider`'s model specification, not on the `ai` runtime. If the AI SDK were ever abandoned, an adapter implementing that interface keeps existing plugins working. + +## Acceptance Criteria + +- [ ] Each phase 1 provider plugin creates a working AI SDK model from a model ID and API key. +- [ ] Startup fails when an admin model names an unregistered provider or a provider without an admin key. +- [ ] Startup fails when more than one admin model is marked default. +- [ ] A user credential's API key is stored encrypted and never returned by the API. +- [ ] Startup fails without a valid `SECRET_KEY`. +- [ ] Resolution uses the user's key when the user has one for the model. +- [ ] Resolution falls back to the admin key for an admin model when one of the user's roles allows it. +- [ ] Resolution fails with `ModelUnavailable` for a model the user cannot use. +- [ ] A model with `roles: ['user']` is available to every signed-in user. +- [ ] An admin model without `roles` fails startup. +- [ ] A user endpoint whose host is a private IP literal, such as `127.0.0.1` or an IPv4-mapped IPv6 address, is refused. +- [ ] A user without an allowed role cannot use an admin model but can use their own keys for it. +- [ ] An admin model naming an undefined role fails startup. +- [ ] A user endpoint resolving to a private address is refused unless the provider allows private networks. +- [ ] A hostname that resolves to a public address when checked and a private one when connecting is still refused. +- [ ] Redirects from a user endpoint are not followed. +- [ ] A value encrypted with a key in `SECRET_KEYS_OLD` still decrypts, and new values use `SECRET_KEY`. +- [ ] A user endpoint's model references use the user's slug. +- [ ] Each lexicon effort value maps to the AI SDK reasoning value in the table. +- [ ] Reasoning from a different provider than the current turn's is not replayed. +- [ ] `GET /api/models` lists the admin models the user may use plus the models from the user's credentials. + +## Files + +- (to be populated during implementation) diff --git a/specs/search.md b/specs/search.md new file mode 100644 index 0000000..3af85e9 --- /dev/null +++ b/specs/search.md @@ -0,0 +1,72 @@ +# Search + +## Summary + +Users can search the text of their own conversations: titles, their messages, and the assistant's replies. Search runs entirely in the browser, over the local SQLite database from the browser-store spec, using SQLite's FTS5 full-text index. The server has no search index and takes no part in searching. Results link to the matching message with a short highlighted snippet. This spec is part of phase 1 but is built after the initial build-out. + +## Motivation + +Chat history is only worth owning if you can find things in it. A PDS has no search API, and the server keeps no copy of chat content, so the index has to live on the user's device. The browser store already holds a copy of the user's chats, so search is an index over data already there. + +## Design + +### Index + +Search adds two FTS5 tables to the browser store's schema and bumps its schema version, which rebuilds the local database on the next load: + +- `message_fts` stores its own copy of each message's `text`, plus the conversation key, author, and message key as unindexed columns. It does not use an external content table, since the `message` table's composite key gives no stable rowid. +- `conversation_fts` indexes conversation titles the same way. + +Both use the `unicode61` tokenizer with diacritics removed, so search behaves the same in any language. Triggers on `message` and `conversation` keep the index in step on insert, update, and delete. + +Only plaintext is indexed: the text parts of messages. Reasoning, tool input and output, and attachment contents are left out. Encrypted messages have no text, so they never appear. + +### Query + +`search(query, { limit, offset })` in the browser-store worker: + +- Words are ANDed, and a quoted phrase matches exactly. +- User input is tokenized and quoted before it reaches FTS5, so no character can be read as query syntax. +- Title matches rank first, then message matches by FTS5's `bm25` relevance, then newest first. +- Each result has the conversation key and title, the message key and role, a snippet of about 30 words from FTS5's `snippet` function, and the message's creation time. + +Snippets mark matches with private-use characters, which the UI turns into highlight elements. Message text is never inserted as HTML. + +### Coverage + +Search covers every conversation the browser store has downloaded. On a new device, results fill in as the background download progresses, and the UI says how many conversations remain to download. + +### Web UI + +A search box above the chat list. Results replace the list while a query is active, and choosing one opens the conversation scrolled to that message. + +## Scope Boundaries + +- No server-side search or index. +- No searching other people's chats, including chats shared with the user. +- No semantic or vector search. +- No searching attachment contents or extracted PDF text. +- No search inside encrypted conversations. + +## Edge Cases and Decisions + +- The index lives in the browser, so the server never builds a copy of chat content for search. +- FTS5 tables store their own copy of the text instead of linking to `message` by rowid, which the composite key cannot keep stable. +- Search is local to each device. Each device builds its own index from the PDS. + +## Acceptance Criteria + +- [ ] A word from a user message finds that message. +- [ ] A word from an assistant reply finds that reply. +- [ ] A word in a conversation title finds the conversation, ranked above message matches. +- [ ] Multiple words match only messages containing all of them. +- [ ] A quoted phrase matches only that exact phrase. +- [ ] Query syntax characters in user input are treated as plain text and never cause an error. +- [ ] Editing or deleting a message updates or removes it from results. +- [ ] Snippets mark the matched words without inserting HTML. +- [ ] Search makes no server request. +- [ ] While the background download is running, the UI shows how many conversations remain. + +## Files + +- (to be populated during implementation) diff --git a/specs/sharing.md b/specs/sharing.md new file mode 100644 index 0000000..77adbb9 --- /dev/null +++ b/specs/sharing.md @@ -0,0 +1,88 @@ +# Sharing + +## Summary + +A conversation's owner can share it read-only with specific people, by handle, or with anyone who has an atproto account. Sharing changes the conversation space's read policy and member list through `com.atproto.simplespace`, so the owner's PDS enforces it for every app, not just ours. A shared conversation opens at `/s//`. The viewer signs in, and the server reads the conversation with the viewer's own space credential, so the owner's PDS decides whether the viewer may see it. Sharing needs spaces on both sides: fallback users can neither share nor view shared chats. + +## Motivation + +Shareable read-only chats were on the original feature list. Letting the owner's PDS enforce access means another app can open a chat shared with you, and a chat stays shared or private however the owner changes it. Reading with the viewer's credential keeps SCN Chat free of its own access rules. Leaving fallback users out keeps the built-in code minimal, since the fallback exists only until PDSs support spaces. + +## Design + +### Share settings + +A conversation has one of three sharing modes: + +- **Private.** Only the owner. This is the default for new conversations. +- **People.** The owner plus a list of members. +- **Public.** Anyone with an atproto account. + +`PUT /api/conversations/:skey/sharing` takes `{ mode, members?: string[] }`, where members are handles or DIDs. Handles are resolved to DIDs with `@atproto/identity`, and an unresolvable handle fails the request with 400, naming it. The server then updates the space through the owner's session: + +- **Private.** `updateSpace` with `readPolicy: #memberListPolicy`, and `removeMember` for every current member. +- **People.** `updateSpace` with `#memberListPolicy`, `putMember` with `read: true, write: false` for each new member, and `removeMember` for anyone dropped. +- **Public.** `updateSpace` with `readPolicy: #publicPolicy`. The member list is kept, so switching back to people restores it. + +The write policy stays `#memberListPolicy` with no writers, so only the owner's writes are ever tracked. + +`GET /api/conversations/:skey/sharing` reads the state back with `getSpace` and `listMembers`, and returns the mode and members with each member's current handle. A change made in another app shows correctly. + +Both routes return 403 with a message saying sharing needs a spaces PDS when the owner's storage mode is `local`. + +### Viewing + +`GET /api/shared/:ownerDid/:skey` requires a signed-in viewer whose storage mode is `space`. Otherwise it returns 403 with a message saying viewing shared chats needs a spaces PDS. + +1. The server gets a delegation token from the viewer's PDS for the conversation space, through the viewer's session. The permission set's read access to conversation spaces of any authority allows this. +2. It exchanges the token for a space credential at the owner's PDS, the space authority, with the same DPoP code chat-storage uses for its own credentials. The owner's PDS applies the read policy and member list. +3. With the credential, it reads the owner's info record with `getRecord` and every message with `listRecords`, from the owner's PDS. +4. It returns the title and messages. The system prompt and the owner's preferences are left out. Tags live in the owner's private settings space, so viewers never see them. Any member can still read the system prompt through the protocol, since it is in the info record, so leaving it out is a presentation choice, not a privacy guarantee. + +A refused credential (`UserNotAuthorized`), a missing space (`SpaceNotFound`), or a deleted one (`SpaceDeleted`) all return 404. A private conversation's existence is therefore not revealed. Viewer credentials are not cached, so revoking access takes effect on the viewer's next page load. + +Attachments in a shared conversation are fetched the same way, with `getBlob` and the viewer's credential, through `GET /api/shared/:ownerDid/:skey/blobs/:cid`. + +### Web UI + +The conversation view has a share control showing the mode and members, with a copyable `/s//` link. It is hidden for fallback users. The shared view shows the conversation with branch navigation, the owner's handle, and no composer, regenerate, or edit controls. + +## Scope Boundaries + +- No sharing or viewing for fallback users. +- No anonymous viewing without an atproto account. +- No write access for members. +- No "shared with me" list. Viewers open shared chats by link. +- No notifications to members when a chat is shared. +- No forking a shared chat into the viewer's own conversations. +- No caching or syncing of other users' conversations. + +## Edge Cases and Decisions + +- Shared views are read with the viewer's own space credential, so the owner's PDS makes every access decision. +- Viewer credentials are minted per request and never cached, trading a little latency for immediate revocation. +- Switching to public keeps the member list, so switching back does not lose it. +- Refused, missing, and deleted conversations all return 404. +- A conversation owned by a user of a different app can be viewed too, as long as its space is a conversation space the viewer can read. + +## Acceptance Criteria + +- [ ] Sharing with people sets a member-list read policy and adds each member with read access and no write access. +- [ ] Removing a person removes their membership. +- [ ] Making a conversation public sets the public read policy and keeps the member list. +- [ ] Making it private again removes every member. +- [ ] An unresolvable handle fails with 400, naming it. +- [ ] Reading the share settings reflects changes made outside the app. +- [ ] A fallback owner gets 403 from both sharing routes. +- [ ] A fallback viewer gets 403 from the shared view. +- [ ] A signed-out request to the shared view returns 401. +- [ ] A member can view a people-shared conversation through their own space credential. +- [ ] Any spaces user can view a public conversation. +- [ ] A refused credential, a missing space, or a deleted space all return 404. +- [ ] After access is revoked, the viewer's next request returns 404. +- [ ] Attachments in a shared conversation load through the viewer's credential. +- [ ] The shared view's response has no system prompt. + +## Files + +- (to be populated during implementation) diff --git a/specs/titles.md b/specs/titles.md new file mode 100644 index 0000000..e8e0c1a --- /dev/null +++ b/specs/titles.md @@ -0,0 +1,80 @@ +# Titles + +## Summary + +After the first reply in a conversation finishes, SCN Chat asks a model for a short title and writes it to the conversation's info record with `titleSource: "generated"`. This is the first turn hook, shipped as the `@scn-chat/plugin-titles` plugin. It never replaces a title the user wrote, and it respects the user's `generateTitles` preference. + +## Motivation + +A chat list of "New chat" entries is useless. Automatic titles are standard in ChatGPT, Claude, and Open WebUI. Building titles as a plugin also proves the hook system works for a real feature. + +## Design + +### Plugin + +`plugins/titles` exports a factory: + +```ts +titles({ model?: ModelRef, maxWords?: number }) +``` + +- `model` picks the model used for titles, for example a cheap admin model. When it is left out, the title uses the same model and credentials that wrote the reply. +- `maxWords` defaults to 6. + +The plugin registers one `turn:after` action. + +### When a title is generated + +The handler generates a title only when all of these are true: + +1. The reply's status is `complete`. +2. The conversation's info record has no `title`. +3. `info.titleSource` is not `"user"`. +4. The user's preferences do not set `generateTitles` to `false`. A missing preferences record counts as `true`. +5. The conversation's content is not encrypted. + +The turn context from the chat-turns spec carries the conversation info and the user's preferences, so the handler makes no extra reads. + +### Generating + +The handler calls `ctx.models.generateText` for the user. The system prompt asks for a title of at most `maxWords` words, with no quotes and no trailing punctuation, in the language of the conversation. The input is the text parts of the triggering user message and the reply, each cut to 2,000 characters. Reasoning, tool calls, and attachments are left out. + +The output is cleaned up before writing. Whitespace is collapsed, surrounding quotes and trailing periods are removed, and the result is cut to the lexicon's 300 graphemes. An empty result is treated as a failure. + +### Writing + +The handler calls `ctx.conversations.updateInfo` with `title` and `titleSource: "generated"`. The storage layer re-reads the info record before writing. If the user has set a title in the meantime, the update is skipped. + +### Failure + +A failed model call or an empty result logs a warning with the conversation URI and the error, and no title is written. A title is a nice-to-have, so a failure never affects the turn. The next completed reply tries again, since the conversation still has no title. + +## Scope Boundaries + +- No title regeneration after the first one succeeds. +- No titles for encrypted conversations. +- No UI beyond showing the title. Renaming a chat is part of the web-ui spec, and sets `titleSource: "user"`. + +## Edge Cases and Decisions + +- The default title model is the one that wrote the reply, so a BYO-key user's titles bill to their own key, and nothing breaks when no cheap admin model is configured. +- A title the user writes while generation is running wins, because the write re-checks `titleSource`. +- Direct-PDS clients get titles too, since the hook runs for every turn the server generates, whatever started it. + +## Acceptance Criteria + +- [ ] A completed first reply in an untitled conversation produces a title with `titleSource: "generated"`. +- [ ] No title is generated when the info record already has a title. +- [ ] No title is generated when `titleSource` is `"user"`, even with no title set. +- [ ] No title is generated when the user's preferences set `generateTitles` to `false`. +- [ ] A missing preferences record allows titles. +- [ ] No title is generated for replies with status `error`, `cancelled`, or `pending`. +- [ ] A configured `model` option is used instead of the reply's model. +- [ ] Quotes, trailing periods, and extra whitespace are removed, and long output is cut to 300 graphemes. +- [ ] A failed model call logs a warning, writes nothing, and the turn stays `complete`. +- [ ] A later completed reply retries after an earlier failure. +- [ ] A user title written during generation is not overwritten. + +## Files + +- (to be populated during implementation) diff --git a/specs/web-ui.md b/specs/web-ui.md new file mode 100644 index 0000000..1a874d5 --- /dev/null +++ b/specs/web-ui.md @@ -0,0 +1,99 @@ +# Web UI + +## Summary + +The web UI is a deliberately bare React app. It exposes every phase 1 feature with plain HTML elements and almost no styling, so a designer can build the real interface on top without fighting existing design decisions. It covers signing in, the chat list, a conversation with branch navigation, the composer with model and effort choice and attachments, streaming replies, sharing, and settings for preferences, API keys, and plugin settings. It talks to the server through Hono's typed client, so API changes surface as type errors. + +## Motivation + +A separate designer will build the production interface. The phase 1 UI exists to prove the features work end to end, and to give the designer working components and data flow to restyle or replace. Every visual choice made here is one the designer might have to undo, so this UI makes as few as possible. + +## Design + +### Stack + +- React with Vite, from the foundation spec. +- `wouter` for routing, a router of about 2 KB. +- `hono/client`'s `hc` with the server's exported route types, for a typed API client with no hand-written API types. +- Chats are read from the browser store, through its worker API. Writes and other data go through the typed client with small hooks. No state management or data-fetching library. +- `EventSource` for reply streams. +- One small stylesheet for layout only: a sidebar and main column with flexbox, and `white-space: pre-wrap` on message text. No colors, fonts, or component library. + +### Routes + +| Route | Screen | +|---|---| +| `/login` | A handle field and a sign-in button that goes to `/oauth/login` | +| `/` | Chat list sidebar, and a new chat | +| `/c/:skey` | Chat list sidebar and a conversation | +| `/s/:ownerDid/:skey` | A shared conversation, read-only | +| `/settings` | Preferences, API keys, and plugin settings | + +Any route except `/login` and a shared route redirects to `/login` when `/api/me` returns 401. A shared route shows a sign-in prompt instead. + +### Components + +- **ChatList.** Conversation titles from the browser store, newest first. Untitled conversations show "New chat". It has a button to start a new chat. +- **Conversation.** It shows the messages on the current branch, from the root to the selected leaf. A message with siblings shows "‹ 2 / 3 ›" controls that switch branches. By default the branch follows the newest sibling at each level. It also has the title, with rename, the share control from the sharing spec, and a sync button that calls the conversation's sync route and then refreshes it in the browser store. +- **Message.** + - Text parts are shown as plain text, with no Markdown rendering. + - Reasoning parts go in a closed `
`. + - Tool calls and results go in a `
` with their JSON. + - Sources are shown as a list of links. + - Images are shown inline, and files as their names. + - A pending reply shows streamed text as it arrives. An errored reply shows its error, and a cancelled reply is labeled as stopped. +- **Message actions.** On user messages, edit, which opens the composer prefilled and sends a sibling. On assistant replies, regenerate, with an optional model change. While generating, stop. +- **Composer.** A textarea, a model select from `/api/models`, an effort select for models with the reasoning capability, and an attach button. The attach button accepts images only for models with vision, plus any type an ingester supports. Attachments upload when chosen and show a progress state. Enter sends, and Shift+Enter adds a newline. +- **Settings.** + - Preferences: default model, default effort, custom instructions, and generate titles. + - API keys: add, list with the last four characters, and delete. For providers with user endpoints, a base URL field. + - Plugin settings: one form per plugin with user settings, generated from its JSON schema. + - Background sync: an on and off switch, shown only when the admin allows users to opt out. + - This device: a button that deletes and rebuilds the browser store. + +### Schema forms + +`SchemaForm` renders a JSON schema object as labeled inputs. It supports strings, as text inputs or password inputs for secret fields, plus numbers, booleans as checkboxes, and string enums as selects. Any other type is shown as unsupported, not guessed at. + +### Streaming + +After sending, the conversation subscribes to the reply's stream endpoint and appends deltas to the pending reply. On the `status` event, it asks the browser store to refresh the conversation, which fetches the final record. If the stream fails, it refreshes the conversation every two seconds until the reply leaves `pending`. + +## Scope Boundaries + +- No visual design, theming, dark mode, icons, or animations. +- No Markdown or code highlighting. +- No PWA, offline support, or notifications. +- No mobile-specific layout beyond what the browser does. +- No search box. That is the search spec. +- No admin screens. + +## Edge Cases and Decisions + +- Message text is plain text, so the designer chooses the Markdown renderer and its sanitizing. +- The typed Hono client comes from the server's route types, so the web app and server share one definition of the API. +- The default branch is the newest sibling at each level, matching how ChatGPT shows the latest regeneration. + +## Acceptance Criteria + +- [ ] Signing in with a handle starts OAuth, and the app returns to the chat list afterwards. +- [ ] Signed-out users are sent to `/login`. +- [ ] Creating a chat, sending a message, and watching the reply stream works end to end. +- [ ] The chat list shows new titles once they are generated. +- [ ] Sibling controls switch branches and show the matching replies. +- [ ] Regenerating adds a sibling reply and selects it. +- [ ] Editing a message adds a sibling user message with its own reply. +- [ ] Stop cancels a generating reply, which then shows as stopped. +- [ ] The effort select appears only for reasoning models. +- [ ] Image attachment is disabled for models without vision. +- [ ] A PDF attachment uploads, shows its name, and reaches the model as text. +- [ ] Settings save preferences, add and delete API keys, and save plugin settings through generated forms. +- [ ] A shared link shows the conversation read-only to a permitted viewer. +- [ ] Reasoning and tool details are collapsed by default. +- [ ] If the stream drops, the reply still appears through repeated refreshes. +- [ ] The sync button syncs the conversation and shows changes written from another client. +- [ ] The background sync switch appears only when the admin allows opting out, and saves the account setting. + +## Files + +- (to be populated during implementation) -- knotmirror 0.0.1