replace big-tech and host basic infrastructure yourself! ordner.fyi
ordner docs developers reference-client.md
20 kB · 513 lines
Markdown
at main

The reference client #

scripts/annotate-example.ts is a complete annotator in one file: it finds an install's authorization server from the JMAP URL alone, registers itself, signs a person in with the device grant, declares a field and a view of its own, annotates every file it has not annotated yet, and keeps doing so as new files arrive. It uses only what Node.js ships with (fetch, node:crypto, node:fs/promises, node:util, node:timers/promises), so nothing in it depends on this repository. Copy it and replace the two stand-ins (§7.2) with real models.

This chapter walks through the file in the order it runs. The functions appear in the file in the same order:

discover → register → signIn → session → defineField → defineView → annotateQueue → follow

1. Running it #

node scripts/annotate-example.ts --jmap https://ordner.example \
  --state ~/.config/example-annotator.json --follow 600
Flag Meaning
--jmap <URL> the install's origin, its /.well-known/jmap, or a JMAP session URL
--issuer <URL> the authorization server, when the JMAP answer does not name it (§2)
--state <file> where the registration and the refresh token are kept between runs; default annotate-state.json
--follow <seconds> how long to keep listening for new files once the queue is empty; 0 (the default) annotates the queue once and exits
--name <text> the client_name the person sees on the approval page; default Example Annotator

It needs Node.js 22.18 or later, which runs TypeScript files directly. It exits 0 when it is done and 1 on any failure, with the reason on standard error.

A first run prints:

registered client rp-YN0dGtWLNdfv
Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV
(or open https://ordner.example/auth/device?user_code=ZC7YJCWV)
approved; granted openid offline_access drive:read drive:annotate drive:faces
field category defined
writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv
view documents defined
annotated IMG_1.JPG (picture, 1 face(s))
annotated notes.txt (document)
annotated report.pdf (document)
queue empty after 3 file(s)
done

A second run with the same --state needs no approval and finds nothing left to do:

using client rp-YN0dGtWLNdfv
signed in with the kept refresh token
field category defined
writing as source 4aa0680d-ff45-448d-971a-e856c5aa0910.app.rp-YN0dGtWLNdfv
view documents defined
queue empty after 0 file(s)
following changes for 600 s
listening for changes
files changed
annotated later.txt (document)
done

2. Discovery #

The client is given one URL and finds everything else from it.

  1. The session, without credentials. An origin alone stands for its /.well-known/jmap (RFC 8620 §2.2). The answer is 401, and its WWW-Authenticate header carries a Bearer challenge with a resource_metadata parameter (RFC 9728 §5.1):

    WWW-Authenticate: Basic realm="mail", charset="UTF-8", Bearer realm="jmap",
      resource_metadata="https://ordner.example/.well-known/oauth-protected-resource"
    

    (one header line, folded here). The client looks for the parameter anywhere in the header.

  2. The protected resource's metadata (RFC 9728 §2) names the authorization server: authorization_servers[0], for example https://ordner.example/auth.

  3. The authorization server's metadata (RFC 8414 §3) names the three endpoints the client uses: registration_endpoint, device_authorization_endpoint and token_endpoint. RFC 8414 §3.1 puts the well-known part between the host and the issuer's path:

    const rfc8414 = `${u.origin}/.well-known/oauth-authorization-server${path}`;
    

    The provider is also an OpenID provider, so <issuer>/.well-known/openid-configuration answers the same document; the client falls back to it. The document's issuer must equal the issuer asked for (RFC 8414 §3.3), or the client stops.

--issuer replaces steps 1 and 2. It is needed only when the JMAP URL given is served without the challenge parameter, for instance a file service reached directly rather than through the install's public origin.

3. Registration #

An application registers itself (RFC 7591 §3.1) once per installation, not once per run: every registration is a client the person approves, sees under Connected apps, and may remove.

body: JSON.stringify({
	client_name: name,
	redirect_uris: ["http://127.0.0.1/callback"],
	token_endpoint_auth_method: "none",
	grant_types: ["authorization_code", "refresh_token", DEVICE_GRANT],
	scope: SCOPE,
}),
  • redirect_uris must be loopback addresses (RFC 8252 §7.3), even for an application that only ever uses the device grant: automatic registration admits native applications only. Any other redirect URI is refused with invalid_redirect_uri.

  • grant_types must include authorization_code; the device grant (urn:ietf:params:oauth:grant-type:device_code) and refresh_token may stand beside it.

  • token_endpoint_auth_method is none: a native application is a public client and holds no secret (RFC 8252 §8.4).

  • scope lists everything the application may ever ask for:

    const SCOPE = "openid offline_access drive:read drive:annotate drive:faces";
    

    drive:read reads every file and everything other applications wrote about it; drive:annotate writes annotations, never content, names or places; drive:faces reads and writes face embeddings; offline_access gives a refresh token. Every data scope needs a read scope beside it, and openid is required.

  • client_name is shown to the person as the name the application gives itself, marked as not checked by anyone. A name that could pass for the install's own (it contains the organisation's name, "admin", "support", "verified", "login" and the like, or mixes scripts) is refused with invalid_client_metadata.

The answer's client_id is kept in the state file, with registration_client_uri and registration_access_token (RFC 7592 §3). On the next run the client reads its registration first (RFC 7592 §2.1):

const res = await fetch(uri, {
	headers: { authorization: `Bearer ${token}` },
});

200 means the registration still stands. 401 means it is gone (an install removes a registration that is never used, and an administrator may ban one): the client forgets its refresh token and registers again.

The state file holds a refresh token, so it is written whole, renamed into place, and readable by its owner only (mode 0600).

4. Signing in with the device grant #

An application without a browser of its own (an agent on a server, a command-line tool) uses the device authorization grant (RFC 8628).

  1. A kept refresh token is tried first (RFC 6749 §6). If the token endpoint answers invalid_grant (the person revoked the application, or removed it), the client forgets it and starts the device grant.

  2. The device authorization request (RFC 8628 §3.1): client_id and scope, form-encoded, to device_authorization_endpoint. The answer carries device_code, user_code, verification_uri, verification_uri_complete, expires_in and interval.

  3. The person is told (RFC 8628 §3.3), in exactly one line, with the complete URI on the next:

    Open https://ordner.example/auth/device and enter the code ZC7Y-JCWV
    (or open https://ordner.example/auth/device?user_code=ZC7YJCWV)
    
  4. The client polls the token endpoint (RFC 8628 §3.4) with grant_type=urn:ietf:params:oauth:grant-type:device_code, waiting interval seconds (5 when absent) before each request, and reads the error codes of RFC 8628 §3.5:

    switch (body.error) {
    	case "authorization_pending":
    		continue;
    	case "slow_down":
    		interval += 5000;
    		continue;
    	case "access_denied":
    		throw new Failure("the person declined");
    	case "expired_token":
    		throw new Failure("the code expired before it was approved; run again");
    

4.1 What the person sees #

The person opens the address on any device and signs in with their passkey. After the code, the approval page shows:

  • the name the application gives itself, with the warning that anyone can register an application under any name;
  • "This app runs on a device or server", in place of the address a browser application would be sent back to;
  • under "If you allow it, the app may:", one line per data scope asked:
    • "See every file and picture in your drive, and everything other apps found in them" (drive:read, always granted with the rest);
    • "Add descriptions, labels, text, faces and album suggestions" (drive:annotate);
    • "Use face data that identifies people (biometric)" (drive:faces);
  • a choice to narrow everything to "Only my Pictures folder, not the rest of my drive";
  • "The ability to stay signed in without asking you again" (offline_access).

Each line but the read one can be unticked. What was ticked is what is granted, and the token answer says so in its scope (RFC 6749 §5.1). The client keeps that list and acts on it: without drive:annotate (or its Pictures-only form, photos:annotate) it stops, and without drive:faces it writes faces without their embeddings (§7.4). A person who narrows the grant to Pictures gets photos:read and photos:annotate in place of the drive scopes; the client then reads and annotates the Pictures folder alone.

The person can withdraw a line or the whole application later under Connected apps. A withdrawn line is gone from the token at once; a revoked application's tokens stop working at once, and its refresh token answers invalid_grant.

4.2 Tokens #

The access token is a Bearer token for the install's JMAP only. It expires after a short time; a JMAP answer of 401 makes the client refresh it once and repeat the request. A refresh may hand out a new refresh token, which replaces the kept one at once.

5. The JMAP session #

The session (RFC 8620 §2), asked for with the token, names the API, upload, download and push URLs, the accounts and the server's limits:

Session member Used for
primaryAccounts["urn:ietf:params:jmap:filenode"] the person's drive: the accountId of every call
capabilities["https://ns.ordner.fyi/svc/jmap/files"] required: annotations, fields, views and vectors
capabilities["https://ns.ordner.fyi/svc/jmap/photos"] optional: pictures' photo property and their faces
capabilities["urn:ietf:params:jmap:core"] maxObjectsInGet, maxObjectsInSet, maxCallsInRequest
apiUrl, uploadUrl, downloadUrl, eventSourceUrl may be relative to the session's URL

The URL templates are RFC 6570 level 1: each {name} is replaced by its value with every character but the unreserved ones percent-encoded (RFC 6570 §3.2.2).

Every request is one of the person's and counts against their request limits. A 429 answer means "wait": the client sleeps as long as Retry-After says (RFC 9110 §10.2.3) and asks again.

6. A field and a view #

6.1 The field #

Beyond a description and labels, an application may declare typed fields of its own. The server indexes them by type, never by meaning:

category: {
	type: "string",
	title: { "x-default": "Category", de: "Kategorie" },
	shown: true,
	facet: true,
	searchable: true,
},

shown puts the value in the file's details in Files, facet offers the field as a filter with counts, searchable puts its text in the full-text search. AnnotationSchema/set replaces the application's whole schema, so every run names every field it uses; running it again changes nothing.

6.2 Its own source #

What an application writes is kept under its source: the person whose credential it is, and the application. The server makes the source from the credential, never from anything the request claims, as <actorId>.app.<clientId>. Two people using the same application are two sources; one person's two installations of it, with two registrations, are two sources too. The client reads its source back from AnnotationSchema/get: the schema whose source.app.id is its own client_id.

6.3 The view #

A view is a saved query that Files lists beside its own places, under the application's name:

v: {
	key: "documents",
	title: { "x-default": "Documents", de: "Dokumente" },
	in: "files",
	layout: "list",
	filter: { field: { source, name: "category", equals: "document" } },
	columns: ["category"],
},

A view is not a folder: it lists what the query answers now, so writing the field keeps it current. key is the application's own name for the view; a create with a key it used before updates that view, so running it again makes no second one. columns name the application's shown fields.

7. The queue #

7.1 Reading a page #

isAnnotated: false is the application's own queue: the files whose current content this source has not annotated, either never or only an older version. Query and get go in one request, the get taking the query's ids by back-reference (RFC 8620 §3.7):

[
	"FileNode/query",
	{ accountId: s.accountId, filter: { isAnnotated: false }, limit },
	"q",
],
[
	"FileNode/get",
	{
		accountId: s.accountId,
		"#ids": { resultOf: "q", name: "FileNode/query", path: "/ids" },
		properties,
	},
	"g",
],

properties are id, name, type, size, blobId, and photo when the Photos capability is served. A page is at most 50 files, and never more than maxObjectsInGet or maxObjectsInSet. The client takes pages until one comes back empty.

The blobId read here is the file's content as this credential reads it. Every write names it, and the server refuses the write with contentChanged when the file has changed since: an application never annotates bytes it did not read.

7.2 Where the models go #

Two functions stand in for real intelligence, each under a banner in the file.

label(file) is where a model goes: it says what a file is. The stand-in looks at the media type only and answers a category (picture, document, audio, video or other), a description, and one label with a confidence of 0.5. A real annotator downloads the content (or a picture's smaller renditions) and runs a classifier or a vision model here.

embed(bytes) is where an embedding model goes: it turns content into a vector. The stand-in makes a 64-dimension unit vector from a SHA-256 of the bytes, so a vector is close only to itself. A real one answers vectors whose closeness means similar content, so that FileNode/query's similarTo finds related files. The model name (example-hash-64@1) says what made the vector and its version; vectors of different models are never compared.

7.3 Text #

For a text/plain file of at most 1 MiB the client downloads the content (RFC 8620 §6.2), and, when it is valid UTF-8, uploads it again (RFC 8620 §6.1) as the annotation's text:

headers: { "content-type": "text/plain; charset=utf-8" },

The annotation names the upload's blobId as text. The text enters the full-text search at once, so Files finds the file by its words. A real annotator writes here what it read out of a PDF, an image or a recording.

7.4 Writing #

All of a page's writes go in one request: FileAnnotation/set with every file's annotation, and FileVector/set with every file's vector.

const update: Obj = {
	blobId,
	description: found.description,
	labels: found.labels,
	fields: { category: found.category },
};

A vector is base64 of little-endian 32-bit floats:

vectors[f.id] = {
	blobId,
	vectors: [
		{
			model: "example-hash-64@1",
			kind: found.category === "picture" ? "image" : "text",
			vector: vectorText(embed(bytes)),
		},
	],
};

A picture the install has read (its photo is not null) also gets one face: its centre and size as fractions of the picture as displayed, and a confidence. The embedding goes with it only when drive:faces was granted; a write carrying an embedding without it is refused as a whole with forbidden.

update.faces = [
	{
		area: { x: 0.5, y: 0.4, w: 0.2, h: 0.25 },
		confidence: 0.5,
		...(mayEmbedFaces
			? { embedding: { model: "example-face-64@1", vector: DUMMY_FACE } }
			: {}),
	},
];

Faces are kept in the picture's XMP sidecar (IMG_1.JPG.xmp), so other photo programs read them. A sidecar is the picture's metadata, not a file to annotate: it never enters the queue, whether a face write made it or another program did. A face list replaces the faces this source found before and that the person has not touched since, so writing it again leaves one face, not two.

7.5 Refusals #

Each file is answered on its own, in notUpdated:

  • contentChanged: the file changed while it was read. Nothing is written; it comes back in the next page with its new blobId.
  • Anything else: the client says why, and on the next page writes the smallest annotation there is, { blobId } alone, which records "read, nothing to say" so that the queue moves on. If even that is refused, the client stops.

8. Following push #

With --follow, the client listens on the JMAP EventSource (RFC 8620 §7.3) for the given time:

const url = expand(c.session.eventSourceUrl, {
	types: "FileNode",
	closeafter: "no",
	ping: "30",
});

It reads the stream with fetch, because a browser's EventSource cannot send an Authorization header, and parses text/event-stream (WHATWG HTML §9.2) itself: event: state events whose data is a JMAP StateChange:

event: state
id: eyI2MjZm…
data: {"@type":"StateChange","changed":{"626f6a28-…":{"FileNode":"f1"}}}

When the account's FileNode state differs from the last one seen, it prints files changed and reads the queue again (§7). The stream tells only of changes from the moment it is open, so on every connection the client also reads the queue once, to catch what changed before. A queue read already running is not started twice; one more follows it.

A dropped connection is opened again after the server's retry (one second when it gave none), with Last-Event-ID set to the last event's id. When the time is up the client finishes the queue read in progress and exits.

The client's own writes change the FileNode state too, so each of its pages is followed by one more queue read, which finds nothing.

9. Checked against the install #

A seam test (seams/src/annotate-example.test.ts) runs this file as a child process against the real authorization and file services on local sockets, approves its device code as the approval page does, and reads back, as the person, what it wrote: every file's annotation under the application's source with its field, the text found by search, the view, the vectors, the face with its embedding, and on a second account the face without one when drive:faces is unticked. A second run with the same state file must reuse the registration, sign in by the refresh token, find an empty queue, and annotate a file added while it follows push. A change to the server that breaks this client fails that test.

Not verified #

  • Discovery from a 401 with resource_metadata is not exercised by the test: it reaches the file service directly and passes --issuer.
  • The slow_down, access_denied, expired_token and 429 paths, the RFC 7592 re-registration after a removed client, and reconnecting a dropped event stream are written to the standards but not run by the test.
  • The Photos-only narrowing is not run with this client.