# The reference client `scripts/annotate-example.ts` is a complete annotator in one file: it finds an install's authorization server from the JMAP URL alone, registers itself, signs a person in with the device grant, declares a field and a view of its own, annotates every file it has not annotated yet, and keeps doing so as new files arrive. It uses only what Node.js ships with (`fetch`, `node:crypto`, `node:fs/promises`, `node:util`, `node:timers/promises`), so nothing in it depends on this repository. Copy it and replace the two stand-ins (§7.2) with real models. This chapter walks through the file in the order it runs. The functions appear in the file in the same order: ``` discover → register → signIn → session → defineField → defineView → annotateQueue → follow ``` ## 1. Running it ``` node scripts/annotate-example.ts --jmap https://ordner.example \ --state ~/.config/example-annotator.json --follow 600 ``` | Flag | Meaning | |---|---| | `--jmap ` | the install's origin, its `/.well-known/jmap`, or a JMAP session URL | | `--issuer ` | the authorization server, when the JMAP answer does not name it (§2) | | `--state ` | where the registration and the refresh token are kept between runs; default `annotate-state.json` | | `--follow ` | how long to keep listening for new files once the queue is empty; `0` (the default) annotates the queue once and exits | | `--name ` | the `client_name` the person sees on the approval page; default `Example Annotator` | It needs Node.js 22.18 or later, which runs TypeScript files directly. It exits 0 when it is done and 1 on any failure, with the reason on standard error. A first run prints: ``` registered client rp-YN0dGtWLNdfv Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV (or open https://ordner.example/auth/device?user_code=ZC7YJCWV) approved; granted openid offline_access drive:read drive:annotate drive:faces field category defined writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv view documents defined annotated IMG_1.JPG (picture, 1 face(s)) annotated notes.txt (document) annotated report.pdf (document) queue empty after 3 file(s) done ``` A second run with the same `--state` needs no approval and finds nothing left to do: ``` using client rp-YN0dGtWLNdfv signed in with the kept refresh token field category defined writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv view documents defined queue empty after 0 file(s) following changes for 600 s listening for changes files changed annotated later.txt (document) done ``` ## 2. Discovery The client is given one URL and finds everything else from it. 1. **The session, without credentials.** An origin alone stands for its `/.well-known/jmap` (RFC 8620 §2.2). The answer is `401`, and its `WWW-Authenticate` header carries a Bearer challenge with a `resource_metadata` parameter (RFC 9728 §5.1): ``` WWW-Authenticate: Basic realm="mail", charset="UTF-8", Bearer realm="jmap", resource_metadata="https://ordner.example/.well-known/oauth-protected-resource" ``` (one header line, folded here). The client looks for the parameter anywhere in the header. 2. **The protected resource's metadata** (RFC 9728 §2) names the authorization server: `authorization_servers[0]`, for example `https://ordner.example/auth`. 3. **The authorization server's metadata** (RFC 8414 §3) names the three endpoints the client uses: `registration_endpoint`, `device_authorization_endpoint` and `token_endpoint`. RFC 8414 §3.1 puts the well-known part between the host and the issuer's path: ```ts const rfc8414 = `${u.origin}/.well-known/oauth-authorization-server${path}`; ``` The provider is also an OpenID provider, so `/.well-known/openid-configuration` answers the same document; the client falls back to it. The document's `issuer` must equal the issuer asked for (RFC 8414 §3.3), or the client stops. `--issuer` replaces steps 1 and 2. It is needed only when the JMAP URL given is served without the challenge parameter, for instance a file service reached directly rather than through the install's public origin. ## 3. Registration An application registers itself (RFC 7591 §3.1) **once per installation**, not once per run: every registration is a client the person approves, sees under Connected apps, and may remove. ```ts body: JSON.stringify({ client_name: name, redirect_uris: ["http://127.0.0.1/callback"], token_endpoint_auth_method: "none", grant_types: ["authorization_code", "refresh_token", DEVICE_GRANT], scope: SCOPE, }), ``` - `redirect_uris` must be loopback addresses (RFC 8252 §7.3), even for an application that only ever uses the device grant: automatic registration admits native applications only. Any other redirect URI is refused with `invalid_redirect_uri`. - `grant_types` must include `authorization_code`; the device grant (`urn:ietf:params:oauth:grant-type:device_code`) and `refresh_token` may stand beside it. - `token_endpoint_auth_method` is `none`: a native application is a public client and holds no secret (RFC 8252 §8.4). - `scope` lists everything the application may ever ask for: ```ts const SCOPE = "openid offline_access drive:read drive:annotate drive:faces"; ``` `drive:read` reads every file and everything other applications wrote about it; `drive:annotate` writes annotations, never content, names or places; `drive:faces` reads and writes face embeddings; `offline_access` gives a refresh token. Every data scope needs a read scope beside it, and `openid` is required. - `client_name` is shown to the person as the name the application gives itself, marked as not checked by anyone. A name that could pass for the install's own (it contains the organisation's name, "admin", "support", "verified", "login" and the like, or mixes scripts) is refused with `invalid_client_metadata`. The answer's `client_id` is kept in the state file, with `registration_client_uri` and `registration_access_token` (RFC 7592 §3). On the next run the client reads its registration first (RFC 7592 §2.1): ```ts const res = await fetch(uri, { headers: { authorization: `Bearer ${token}` }, }); ``` `200` means the registration still stands. `401` means it is gone (an install removes a registration that is never used, and an administrator may ban one): the client forgets its refresh token and registers again. The state file holds a refresh token, so it is written whole, renamed into place, and readable by its owner only (mode `0600`). ## 4. Signing in with the device grant An application without a browser of its own (an agent on a server, a command-line tool) uses the device authorization grant (RFC 8628). 1. **A kept refresh token is tried first** (RFC 6749 §6). If the token endpoint answers `invalid_grant` (the person revoked the application, or removed it), the client forgets it and starts the device grant. 2. **The device authorization request** (RFC 8628 §3.1): `client_id` and `scope`, form-encoded, to `device_authorization_endpoint`. The answer carries `device_code`, `user_code`, `verification_uri`, `verification_uri_complete`, `expires_in` and `interval`. 3. **The person is told** (RFC 8628 §3.3), in exactly one line, with the complete URI on the next: ``` Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV (or open https://ordner.example/auth/device?user_code=ZC7YJCWV) ``` 4. **The client polls the token endpoint** (RFC 8628 §3.4) with `grant_type=urn:ietf:params:oauth:grant-type:device_code`, waiting `interval` seconds (5 when absent) before each request, and reads the error codes of RFC 8628 §3.5: ```ts switch (body.error) { case "authorization_pending": continue; case "slow_down": interval += 5000; continue; case "access_denied": throw new Failure("the person declined"); case "expired_token": throw new Failure("the code expired before it was approved; run again"); ``` ### 4.1 What the person sees The person opens the address on any device and signs in with their passkey. After the code, the approval page shows: - the name the application gives itself, with the warning that anyone can register an application under any name; - "This app runs on a device or server", in place of the address a browser application would be sent back to; - under "If you allow it, the app may:", one line per data scope asked: - "See every file and picture in your drive, and everything other apps found in them" (`drive:read`, always granted with the rest); - "Add descriptions, labels, text, faces and album suggestions" (`drive:annotate`); - "Use face data that identifies people (biometric)" (`drive:faces`); - a choice to narrow everything to "Only my Pictures folder, not the rest of my drive"; - "The ability to stay signed in without asking you again" (`offline_access`). Each line but the read one can be unticked. What was ticked is what is granted, and the token answer says so in its `scope` (RFC 6749 §5.1). The client keeps that list and acts on it: without `drive:annotate` (or its Pictures-only form, `photos:annotate`) it stops, and without `drive:faces` it writes faces without their embeddings (§7.4). A person who narrows the grant to Pictures gets `photos:read` and `photos:annotate` in place of the drive scopes; the client then reads and annotates the `Pictures` folder alone. The person can withdraw a line or the whole application later under Connected apps. A withdrawn line is gone from the token at once; a revoked application's tokens stop working at once, and its refresh token answers `invalid_grant`. ### 4.2 Tokens The access token is a Bearer token for the install's JMAP only. It expires after a short time; a JMAP answer of `401` makes the client refresh it once and repeat the request. A refresh may hand out a new refresh token, which replaces the kept one at once. ## 5. The JMAP session The session (RFC 8620 §2), asked for with the token, names the API, upload, download and push URLs, the accounts and the server's limits: | Session member | Used for | |---|---| | `primaryAccounts["urn:ietf:params:jmap:filenode"]` | the person's drive: the `accountId` of every call | | `capabilities["https://ns.ordner.fyi/svc/jmap/files"]` | required: annotations, fields, views and vectors | | `capabilities["https://ns.ordner.fyi/svc/jmap/photos"]` | optional: pictures' `photo` property and their faces | | `capabilities["urn:ietf:params:jmap:core"]` | `maxObjectsInGet`, `maxObjectsInSet`, `maxCallsInRequest` | | `apiUrl`, `uploadUrl`, `downloadUrl`, `eventSourceUrl` | may be relative to the session's URL | The URL templates are RFC 6570 level 1: each `{name}` is replaced by its value with every character but the unreserved ones percent-encoded (RFC 6570 §3.2.2). Every request is one of the person's and counts against their request limits. A `429` answer means "wait": the client sleeps as long as `Retry-After` says (RFC 9110 §10.2.3) and asks again. ## 6. A field and a view ### 6.1 The field Beyond a description and labels, an application may declare typed fields of its own. The server indexes them by type, never by meaning: ```ts category: { type: "string", title: { "x-default": "Category", de: "Kategorie" }, shown: true, facet: true, searchable: true, }, ``` `shown` puts the value in the file's details in Files, `facet` offers the field as a filter with counts, `searchable` puts its text in the full-text search. `AnnotationSchema/set` replaces the application's whole schema, so every run names every field it uses; running it again changes nothing. ### 6.2 Its own source What an application writes is kept under its **source**: the person whose credential it is, and the application. The server makes the source from the credential, never from anything the request claims, as `.app.`. Two people using the same application are two sources; one person's two installations of it, with two registrations, are two sources too. The client reads its source back from `AnnotationSchema/get`: the schema whose `source.app.id` is its own `client_id`. ### 6.3 The view A view is a saved query that Files lists beside its own places, under the application's name: ```ts v: { key: "documents", title: { "x-default": "Documents", de: "Dokumente" }, in: "files", layout: "list", filter: { field: { source, name: "category", equals: "document" } }, columns: ["category"], }, ``` A view is not a folder: it lists what the query answers now, so writing the field keeps it current. `key` is the application's own name for the view; a create with a key it used before updates that view, so running it again makes no second one. `columns` name the application's `shown` fields. ## 7. The queue ### 7.1 Reading a page `isAnnotated: false` is the application's own queue: the files whose current content this source has not annotated, either never or only an older version. Query and get go in one request, the get taking the query's ids by back-reference (RFC 8620 §3.7): ```ts [ "FileNode/query", { accountId: s.accountId, filter: { isAnnotated: false }, limit }, "q", ], [ "FileNode/get", { accountId: s.accountId, "#ids": { resultOf: "q", name: "FileNode/query", path: "/ids" }, properties, }, "g", ], ``` `properties` are `id`, `name`, `type`, `size`, `blobId`, and `photo` when the Photos capability is served. A page is at most 50 files, and never more than `maxObjectsInGet` or `maxObjectsInSet`. The client takes pages until one comes back empty. The `blobId` read here is the file's content as this credential reads it. Every write names it, and the server refuses the write with `contentChanged` when the file has changed since: an application never annotates bytes it did not read. ### 7.2 Where the models go Two functions stand in for real intelligence, each under a banner in the file. `label(file)` is **where a model goes**: it says what a file is. The stand-in looks at the media type only and answers a category (`picture`, `document`, `audio`, `video` or `other`), a description, and one label with a confidence of 0.5. A real annotator downloads the content (or a picture's smaller renditions) and runs a classifier or a vision model here. `embed(bytes)` is **where an embedding model goes**: it turns content into a vector. The stand-in makes a 64-dimension unit vector from a SHA-256 of the bytes, so a vector is close only to itself. A real one answers vectors whose closeness means similar content, so that `FileNode/query`'s `similarTo` finds related files. The model name (`example-hash-64@1`) says what made the vector and its version; vectors of different models are never compared. ### 7.3 Text For a `text/plain` file of at most 1 MiB the client downloads the content (RFC 8620 §6.2), and, when it is valid UTF-8, uploads it again (RFC 8620 §6.1) as the annotation's text: ```ts headers: { "content-type": "text/plain; charset=utf-8" }, ``` The annotation names the upload's `blobId` as `text`. The text enters the full-text search at once, so Files finds the file by its words. A real annotator writes here what it read out of a PDF, an image or a recording. ### 7.4 Writing All of a page's writes go in one request: `FileAnnotation/set` with every file's annotation, and `FileVector/set` with every file's vector. ```ts const update: Obj = { blobId, description: found.description, labels: found.labels, fields: { category: found.category }, }; ``` A vector is base64 of little-endian 32-bit floats: ```ts vectors[f.id] = { blobId, vectors: [ { model: "example-hash-64@1", kind: found.category === "picture" ? "image" : "text", vector: vectorText(embed(bytes)), }, ], }; ``` A picture the install has read (its `photo` is not `null`) also gets one face: its centre and size as fractions of the picture as displayed, and a confidence. The embedding goes with it only when `drive:faces` was granted; a write carrying an embedding without it is refused as a whole with `forbidden`. ```ts update.faces = [ { area: { x: 0.5, y: 0.4, w: 0.2, h: 0.25 }, confidence: 0.5, ...(mayEmbedFaces ? { embedding: { model: "example-face-64@1", vector: DUMMY_FACE } } : {}), }, ]; ``` Faces are kept in the picture's XMP sidecar (`IMG_1.JPG.xmp`), so other photo programs read them. A sidecar is the picture's metadata, not a file to annotate: it never enters the queue, whether a face write made it or another program did. A face list replaces the faces this source found before and that the person has not touched since, so writing it again leaves one face, not two. ### 7.5 Refusals Each file is answered on its own, in `notUpdated`: - `contentChanged`: the file changed while it was read. Nothing is written; it comes back in the next page with its new `blobId`. - Anything else: the client says why, and on the next page writes the smallest annotation there is, `{ blobId }` alone, which records "read, nothing to say" so that the queue moves on. If even that is refused, the client stops. ## 8. Following push With `--follow`, the client listens on the JMAP EventSource (RFC 8620 §7.3) for the given time: ```ts const url = expand(c.session.eventSourceUrl, { types: "FileNode", closeafter: "no", ping: "30", }); ``` It reads the stream with `fetch`, because a browser's `EventSource` cannot send an `Authorization` header, and parses `text/event-stream` (WHATWG HTML §9.2) itself: `event: state` events whose `data` is a JMAP `StateChange`: ``` event: state id: eyI2MjZm… data: {"@type":"StateChange","changed":{"626f6a28-…":{"FileNode":"f1"}}} ``` When the account's `FileNode` state differs from the last one seen, it prints `files changed` and reads the queue again (§7). The stream tells only of changes from the moment it is open, so on every connection the client also reads the queue once, to catch what changed before. A queue read already running is not started twice; one more follows it. A dropped connection is opened again after the server's `retry` (one second when it gave none), with `Last-Event-ID` set to the last event's `id`. When the time is up the client finishes the queue read in progress and exits. The client's own writes change the `FileNode` state too, so each of its pages is followed by one more queue read, which finds nothing. ## 9. Checked against the install A seam test (`seams/src/annotate-example.test.ts`) runs this file as a child process against the real authorization and file services on local sockets, approves its device code as the approval page does, and reads back, as the person, what it wrote: every file's annotation under the application's source with its field, the text found by search, the view, the vectors, the face with its embedding, and on a second account the face without one when `drive:faces` is unticked. A second run with the same state file must reuse the registration, sign in by the refresh token, find an empty queue, and annotate a file added while it follows push. A change to the server that breaks this client fails that test. ## Not verified - Discovery from a `401` with `resource_metadata` is not exercised by the test: it reaches the file service directly and passes `--issuer`. - The `slow_down`, `access_denied`, `expired_token` and `429` paths, the RFC 7592 re-registration after a removed client, and reconnecting a dropped event stream are written to the standards but not run by the test. - The Photos-only narrowing is not run with this client.