The reference client #
scripts/annotate-example.ts is a complete annotator in one file: it finds
an install's authorization server from the JMAP URL alone, registers itself,
signs a person in with the device grant, declares a field and a view of its
own, annotates every file it has not annotated yet, and keeps doing so as
new files arrive. It uses only what Node.js ships with (fetch,
node:crypto, node:fs/promises, node:util, node:timers/promises), so
nothing in it depends on this repository. Copy it and replace the two
stand-ins (§7.2) with real models.
This chapter walks through the file in the order it runs. The functions appear in the file in the same order:
discover → register → signIn → session → defineField → defineView → annotateQueue → follow
1. Running it #
node scripts/annotate-example.ts --jmap https://ordner.example \
--state ~/.config/example-annotator.json --follow 600
| Flag | Meaning |
|---|---|
--jmap <URL> |
the install's origin, its /.well-known/jmap, or a JMAP session URL |
--issuer <URL> |
the authorization server, when the JMAP answer does not name it (§2) |
--state <file> |
where the registration and the refresh token are kept between runs; default annotate-state.json |
--follow <seconds> |
how long to keep listening for new files once the queue is empty; 0 (the default) annotates the queue once and exits |
--name <text> |
the client_name the person sees on the approval page; default Example Annotator |
It needs Node.js 22.18 or later, which runs TypeScript files directly. It exits 0 when it is done and 1 on any failure, with the reason on standard error.
A first run prints:
registered client rp-YN0dGtWLNdfv
Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV
(or open https://ordner.example/auth/device?user_code=ZC7YJCWV)
approved; granted openid offline_access drive:read drive:annotate drive:faces
field category defined
writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv
view documents defined
annotated IMG_1.JPG (picture, 1 face(s))
annotated notes.txt (document)
annotated report.pdf (document)
queue empty after 3 file(s)
done
A second run with the same --state needs no approval and finds nothing
left to do:
using client rp-YN0dGtWLNdfv
signed in with the kept refresh token
field category defined
writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv
view documents defined
queue empty after 0 file(s)
following changes for 600 s
listening for changes
files changed
annotated later.txt (document)
done
2. Discovery #
The client is given one URL and finds everything else from it.
-
The session, without credentials. An origin alone stands for its
/.well-known/jmap(RFC 8620 §2.2). The answer is401, and itsWWW-Authenticateheader carries a Bearer challenge with aresource_metadataparameter (RFC 9728 §5.1):WWW-Authenticate: Basic realm="mail", charset="UTF-8", Bearer realm="jmap", resource_metadata="https://ordner.example/.well-known/oauth-protected-resource"(one header line, folded here). The client looks for the parameter anywhere in the header.
-
The protected resource's metadata (RFC 9728 §2) names the authorization server:
authorization_servers[0], for examplehttps://ordner.example/auth. -
The authorization server's metadata (RFC 8414 §3) names the three endpoints the client uses:
registration_endpoint,device_authorization_endpointandtoken_endpoint. RFC 8414 §3.1 puts the well-known part between the host and the issuer's path:const rfc8414 = `${u.origin}/.well-known/oauth-authorization-server${path}`;The provider is also an OpenID provider, so
<issuer>/.well-known/openid-configurationanswers the same document; the client falls back to it. The document'sissuermust equal the issuer asked for (RFC 8414 §3.3), or the client stops.
--issuer replaces steps 1 and 2. It is needed only when the JMAP URL
given is served without the challenge parameter, for instance a file
service reached directly rather than through the install's public origin.
3. Registration #
An application registers itself (RFC 7591 §3.1) once per installation, not once per run: every registration is a client the person approves, sees under Connected apps, and may remove.
body: JSON.stringify({
client_name: name,
redirect_uris: ["http://127.0.0.1/callback"],
token_endpoint_auth_method: "none",
grant_types: ["authorization_code", "refresh_token", DEVICE_GRANT],
scope: SCOPE,
}),
-
redirect_urismust be loopback addresses (RFC 8252 §7.3), even for an application that only ever uses the device grant: automatic registration admits native applications only. Any other redirect URI is refused withinvalid_redirect_uri. -
grant_typesmust includeauthorization_code; the device grant (urn:ietf:params:oauth:grant-type:device_code) andrefresh_tokenmay stand beside it. -
token_endpoint_auth_methodisnone: a native application is a public client and holds no secret (RFC 8252 §8.4). -
scopelists everything the application may ever ask for:const SCOPE = "openid offline_access drive:read drive:annotate drive:faces";drive:readreads every file and everything other applications wrote about it;drive:annotatewrites annotations, never content, names or places;drive:facesreads and writes face embeddings;offline_accessgives a refresh token. Every data scope needs a read scope beside it, andopenidis required. -
client_nameis shown to the person as the name the application gives itself, marked as not checked by anyone. A name that could pass for the install's own (it contains the organisation's name, "admin", "support", "verified", "login" and the like, or mixes scripts) is refused withinvalid_client_metadata.
The answer's client_id is kept in the state file, with
registration_client_uri and registration_access_token (RFC 7592 §3).
On the next run the client reads its registration first (RFC 7592 §2.1):
const res = await fetch(uri, {
headers: { authorization: `Bearer ${token}` },
});
200 means the registration still stands. 401 means it is gone (an
install removes a registration that is never used, and an administrator may
ban one): the client forgets its refresh token and registers again.
The state file holds a refresh token, so it is written whole, renamed into
place, and readable by its owner only (mode 0600).
4. Signing in with the device grant #
An application without a browser of its own (an agent on a server, a command-line tool) uses the device authorization grant (RFC 8628).
-
A kept refresh token is tried first (RFC 6749 §6). If the token endpoint answers
invalid_grant(the person revoked the application, or removed it), the client forgets it and starts the device grant. -
The device authorization request (RFC 8628 §3.1):
client_idandscope, form-encoded, todevice_authorization_endpoint. The answer carriesdevice_code,user_code,verification_uri,verification_uri_complete,expires_inandinterval. -
The person is told (RFC 8628 §3.3), in exactly one line, with the complete URI on the next:
Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV (or open https://ordner.example/auth/device?user_code=ZC7YJCWV) -
The client polls the token endpoint (RFC 8628 §3.4) with
grant_type=urn:ietf:params:oauth:grant-type:device_code, waitingintervalseconds (5 when absent) before each request, and reads the error codes of RFC 8628 §3.5:switch (body.error) { case "authorization_pending": continue; case "slow_down": interval += 5000; continue; case "access_denied": throw new Failure("the person declined"); case "expired_token": throw new Failure("the code expired before it was approved; run again");
4.1 What the person sees #
The person opens the address on any device and signs in with their passkey. After the code, the approval page shows:
- the name the application gives itself, with the warning that anyone can register an application under any name;
- "This app runs on a device or server", in place of the address a browser application would be sent back to;
- under "If you allow it, the app may:", one line per data scope asked:
- "See every file and picture in your drive, and everything other apps
found in them" (
drive:read, always granted with the rest); - "Add descriptions, labels, text, faces and album suggestions"
(
drive:annotate); - "Use face data that identifies people (biometric)" (
drive:faces);
- "See every file and picture in your drive, and everything other apps
found in them" (
- a choice to narrow everything to "Only my Pictures folder, not the rest of my drive";
- "The ability to stay signed in without asking you again"
(
offline_access).
Each line but the read one can be unticked. What was ticked is what is
granted, and the token answer says so in its scope (RFC 6749 §5.1). The
client keeps that list and acts on it: without drive:annotate (or its
Pictures-only form, photos:annotate) it stops,
and without drive:faces it writes faces without their embeddings (§7.4).
A person who narrows the grant to Pictures gets photos:read and
photos:annotate in place of the drive scopes; the client then reads and
annotates the Pictures folder alone.
The person can withdraw a line or the whole application later under
Connected apps. A withdrawn line is gone from the token at once; a revoked
application's tokens stop working at once, and its refresh token answers
invalid_grant.
4.2 Tokens #
The access token is a Bearer token for the install's JMAP only. It expires
after a short time; a JMAP answer of 401 makes the client refresh it once
and repeat the request. A refresh may hand out a new refresh token, which
replaces the kept one at once.
5. The JMAP session #
The session (RFC 8620 §2), asked for with the token, names the API, upload, download and push URLs, the accounts and the server's limits:
| Session member | Used for |
|---|---|
primaryAccounts["urn:ietf:params:jmap:filenode"] |
the person's drive: the accountId of every call |
capabilities["https://ns.ordner.fyi/svc/jmap/files"] |
required: annotations, fields, views and vectors |
capabilities["https://ns.ordner.fyi/svc/jmap/photos"] |
optional: pictures' photo property and their faces |
capabilities["urn:ietf:params:jmap:core"] |
maxObjectsInGet, maxObjectsInSet, maxCallsInRequest |
apiUrl, uploadUrl, downloadUrl, eventSourceUrl |
may be relative to the session's URL |
The URL templates are RFC 6570 level 1: each {name} is replaced by its
value with every character but the unreserved ones percent-encoded
(RFC 6570 §3.2.2).
Every request is one of the person's and counts against their request
limits. A 429 answer means "wait": the client sleeps as long as
Retry-After says (RFC 9110 §10.2.3) and asks again.
6. A field and a view #
6.1 The field #
Beyond a description and labels, an application may declare typed fields of its own. The server indexes them by type, never by meaning:
category: {
type: "string",
title: { "x-default": "Category", de: "Kategorie" },
shown: true,
facet: true,
searchable: true,
},
shown puts the value in the file's details in Files, facet offers the
field as a filter with counts, searchable puts its text in the full-text
search. AnnotationSchema/set replaces the application's whole schema, so
every run names every field it uses; running it again changes nothing.
6.2 Its own source #
What an application writes is kept under its source: the person whose
credential it is, and the application. The server makes the source from the
credential, never from anything the request claims, as
<actorId>.app.<clientId>. Two people using the same application are two
sources; one person's two installations of it, with two registrations, are
two sources too. The client reads its source back from
AnnotationSchema/get: the schema whose source.app.id is its own
client_id.
6.3 The view #
A view is a saved query that Files lists beside its own places, under the application's name:
v: {
key: "documents",
title: { "x-default": "Documents", de: "Dokumente" },
in: "files",
layout: "list",
filter: { field: { source, name: "category", equals: "document" } },
columns: ["category"],
},
A view is not a folder: it lists what the query answers now, so writing the
field keeps it current. key is the application's own name for the view;
a create with a key it used before updates that view, so running it again
makes no second one. columns name the application's shown fields.
7. The queue #
7.1 Reading a page #
isAnnotated: false is the application's own queue: the files whose current
content this source has not annotated, either never or only an older
version. Query and get go in one request, the get taking the query's ids by
back-reference (RFC 8620 §3.7):
[
"FileNode/query",
{ accountId: s.accountId, filter: { isAnnotated: false }, limit },
"q",
],
[
"FileNode/get",
{
accountId: s.accountId,
"#ids": { resultOf: "q", name: "FileNode/query", path: "/ids" },
properties,
},
"g",
],
properties are id, name, type, size, blobId, and photo when
the Photos capability is served. A page is at most 50 files, and never more
than maxObjectsInGet or maxObjectsInSet. The client takes pages until
one comes back empty.
The blobId read here is the file's content as this credential reads it.
Every write names it, and the server refuses the write with
contentChanged when the file has changed since: an application never
annotates bytes it did not read.
7.2 Where the models go #
Two functions stand in for real intelligence, each under a banner in the file.
label(file) is where a model goes: it says what a file is. The stand-in
looks at the media type only and answers a category (picture, document,
audio, video or other), a description, and one label with a confidence
of 0.5. A real annotator downloads the content (or a picture's smaller
renditions) and runs a classifier or a vision model here.
embed(bytes) is where an embedding model goes: it turns content into a
vector. The stand-in makes a 64-dimension unit vector from a SHA-256 of the
bytes, so a vector is close only to itself. A real one answers vectors whose
closeness means similar content, so that FileNode/query's similarTo
finds related files. The model name (example-hash-64@1) says what made the
vector and its version; vectors of different models are never compared.
7.3 Text #
For a text/plain file of at most 1 MiB the client downloads the content
(RFC 8620 §6.2), and, when it is valid UTF-8, uploads it again (RFC 8620
§6.1) as the annotation's text:
headers: { "content-type": "text/plain; charset=utf-8" },
The annotation names the upload's blobId as text. The text enters the
full-text search at once, so Files finds the file by its words. A real
annotator writes here what it read out of a PDF, an image or a recording.
7.4 Writing #
All of a page's writes go in one request: FileAnnotation/set with every
file's annotation, and FileVector/set with every file's vector.
const update: Obj = {
blobId,
description: found.description,
labels: found.labels,
fields: { category: found.category },
};
A vector is base64 of little-endian 32-bit floats:
vectors[f.id] = {
blobId,
vectors: [
{
model: "example-hash-64@1",
kind: found.category === "picture" ? "image" : "text",
vector: vectorText(embed(bytes)),
},
],
};
A picture the install has read (its photo is not null) also gets one
face: its centre and size as fractions of the picture as displayed, and a
confidence. The embedding goes with it only when drive:faces was granted;
a write carrying an embedding without it is refused as a whole with
forbidden.
update.faces = [
{
area: { x: 0.5, y: 0.4, w: 0.2, h: 0.25 },
confidence: 0.5,
...(mayEmbedFaces
? { embedding: { model: "example-face-64@1", vector: DUMMY_FACE } }
: {}),
},
];
Faces are kept in the picture's XMP sidecar (IMG_1.JPG.xmp), so other
photo programs read them. A sidecar is the picture's metadata, not a file
to annotate: it never enters the queue, whether a face write made it or
another program did. A face list replaces the faces this
source found before and that the person has not touched since, so writing
it again leaves one face, not two.
7.5 Refusals #
Each file is answered on its own, in notUpdated:
contentChanged: the file changed while it was read. Nothing is written; it comes back in the next page with its newblobId.- Anything else: the client says why, and on the next page writes the
smallest annotation there is,
{ blobId }alone, which records "read, nothing to say" so that the queue moves on. If even that is refused, the client stops.
8. Following push #
With --follow, the client listens on the JMAP EventSource (RFC 8620 §7.3)
for the given time:
const url = expand(c.session.eventSourceUrl, {
types: "FileNode",
closeafter: "no",
ping: "30",
});
It reads the stream with fetch, because a browser's EventSource cannot
send an Authorization header, and parses text/event-stream (WHATWG HTML
§9.2) itself: event: state events whose data is a JMAP StateChange:
event: state
id: eyI2MjZm…
data: {"@type":"StateChange","changed":{"626f6a28-…":{"FileNode":"f1"}}}
When the account's FileNode state differs from the last one seen, it
prints files changed and reads the queue again (§7). The stream tells only
of changes from the moment it is open, so on every connection the client
also reads the queue once, to catch what changed before. A queue read
already running is not started twice; one more follows it.
A dropped connection is opened again after the server's retry (one second
when it gave none), with Last-Event-ID set to the last event's id. When
the time is up the client finishes the queue read in progress and exits.
The client's own writes change the FileNode state too, so each of its
pages is followed by one more queue read, which finds nothing.
9. Checked against the install #
A seam test (seams/src/annotate-example.test.ts) runs this file as a
child process against the real authorization and file services on local
sockets, approves its device code as the approval page does, and reads back,
as the person, what it wrote: every file's annotation under the
application's source with its field, the text found by search, the view,
the vectors, the face with its embedding, and on a second account the face
without one when drive:faces is unticked. A second run with the same state
file must reuse the registration, sign in by the refresh token, find an
empty queue, and annotate a file added while it follows push. A change to
the server that breaks this client fails that test.
Not verified #
- Discovery from a
401withresource_metadatais not exercised by the test: it reaches the file service directly and passes--issuer. - The
slow_down,access_denied,expired_tokenand429paths, the RFC 7592 re-registration after a removed client, and reconnecting a dropped event stream are written to the standards but not run by the test. - The Photos-only narrowing is not run with this client.