# @nibling/api

The one stateless Cloudflare Worker nibling runs (D22). It is the app's read path: a gateway
that answers repo reads, identity, backlinks, and git fetches from a cache. The app works without
it, reading each service directly, with no global feed, no trending, no link cards, and nothing
from `dev.nibling.actor.getKnown`. All project data, likes, follows, and profiles live on each
account's own PDS. This worker never holds that data as the source of truth.

## Routes

- `GET /xrpc/dev.nibling.feed.getRecent?limit=&cursor=&tag=` — recently created or updated
  projects, newest first. A project sorts by its `updatedAt` (or `createdAt`), clamped to when
  the index saw that version, so a far-future date can't pin it to the top. Like and fork counts
  count distinct accounts.
- `GET /xrpc/dev.nibling.feed.getTrending?limit=&cursor=` — projects ranked by likes and forks
  from the last 7 days, decayed linearly to zero at the window edge. A like or fork counts from
  its `createdAt`, but never later than when the index first saw it, so a post-dated record
  can't stay fresh.
- `GET /xrpc/dev.nibling.actor.getKnown?dids=` — which of the given DIDs have a
  `dev.nibling.project` or `dev.nibling.actor.profile` record. Used to narrow imported Bluesky
  or Tangled follows to accounts that already use nibling.
- `GET /:actor/:name` — a project page: Open Graph tags for crawlers, then a
  `<meta http-equiv="refresh">` redirect to the app for everyone else. `:name` is the project's
  slug or its rkey; a slug is found by listing the author's PDS. Links use the slug when there is one.
- `GET /:actor/:name/embed` — the same metadata, but the page itself is an iframe-friendly shell
  around the app's `/:actor/:name/embed` (nothing navigates away).
- `GET /@:actor/:name` and `/@:actor/:name/embed` — a 301 to the same path without the `@`.
- `GET /oembed?url=https://nibling.dev/actor/name` — oEmbed JSON (`type: "rich"`) for the page
  above.
- `GET /oauth/client-metadata.json`, the atproto OAuth client metadata document (D19), built by `@nibling/sdk/metadata` so the app and the worker share one scope string.

The XRPC routes send `Access-Control-Allow-Origin: *` and a short `Cache-Control`, and error as
`{ "error": "...", "message": "..." }`.

## Gateway

Every read the app makes goes through these routes (`src/gateway/`), all `GET` unless noted, all
with `Access-Control-Allow-Origin: *`. `OPTIONS` on any path answers a preflight that allows
`content-type`, `git-protocol`, and `x-nib-fresh`.

| Route | Upstream | Cached for |
|---|---|---|
| `com.atproto.repo.getRecord` with `cid` | the repo's PDS | 1 year, and only when the answer carries that `cid` |
| `com.atproto.repo.getRecord` without `cid`, `listRecords`, `describeRepo` | the repo's PDS | 15 s, then served stale for 60 s while it refreshes |
| `com.atproto.sync.getBlob` | the repo's PDS | 1 year |
| `blue.microcosm.identity.resolveMiniDoc` | Slingshot, then PLC or did:web and DNS-over-HTTPS | 5 minutes |
| `com.atproto.identity.resolveDid` | PLC or the did:web host | 5 minutes |
| `blue.microcosm.links.getBacklinks`, `getBacklinksCount` | Constellation | 15 s |
| `blue.microcosm.repo.getRecordByUri` (`at_uri`, optional `cid`) | Slingshot | 15 s with the stale window, or 1 year with `cid` |
| `/git/<repo DID>/info/refs?service=git-upload-pack` | the repo's `#tangled_knot` | never |
| `POST /git/<repo DID>/git-upload-pack` | the repo's `#tangled_knot` | 1 year when the body is only `want`, `deepen`, and `done` lines, keyed by its SHA-256; otherwise never |

- Every response says how it was served in `x-nib-cache`: `hit`, `stale`, `miss`, or `bypass`.
  A request with `x-nib-fresh: 1` skips the cached answer and stores the new one, which is how a
  writer reads their own write. The SDK sends it for the signed-in account's own repo.
- The Cache API is per data center, and a purge does not reach the others, so nothing purges.
  Every entry ages out on its TTL. The Cache API ignores `stale-while-revalidate`, so
  `src/gateway/cache.ts` serves stale entries itself and refreshes them with `waitUntil`. Only a
  `200` is stored.
- Each route checks its params at the boundary (a DID or handle through
  `@nibling/sdk/identifier`, an NSID, a record key, a CID, a limit from 1 to 100) and drops any
  other param. A `repo` handle is resolved and forwarded as its DID, so the handle and the DID share
  one cache entry.
- The PDS or knot comes from cached identity and must be a public `https:` URL (`isPublicHttpsUrl`).
  Redirects are followed by hand, at most 3, and each hop is checked the same way. The git routes
  proxy only to the knot the repo DID's document names, never to a host from the request.
- Upstream calls time out after 10 seconds (60 for git). A JSON answer or a blob over 5 MB, and a
  pack over 100 MB, is cut off. An upstream 5xx, timeout, or cut-off becomes a 502 or 504, which
  tells the SDK to retry on the direct path. A 4xx passes through as an answer.
- Blobs are served with `Content-Security-Policy: default-src 'none'; sandbox` and
  `X-Content-Type-Options: nosniff`, since an uploader picks the content type.
- `git-receive-pack` is refused with a 403. Pushes, like PDS writes, go from the browser straight
  to the knot or PDS: OAuth's DPoP tokens are bound to a browser key, and a push token is
  single-use and expires in 60 seconds.

## User-Agent

Every request this worker makes carries
`User-Agent: nibling.dev/<version> (+https://nibling.dev/contact)`, so a service can tell us about
runaway traffic. Set `CONTACT_URL` to change the address. `upstream()` (`src/upstream.ts`) adds the
header, and only it makes the branded `Upstream` type that the router, gateway, backfill, ingest,
and identity code accept, so a bare `fetch` does not typecheck. Jetstream is opened with a `fetch`
carrying `Upgrade: websocket` rather than `new WebSocket`, so its handshake carries the header
too. `src/services.ts` is the only file that reads `globalThis.fetch`. `tests/user-agent.test.ts`
replaces the global `fetch` while link cards, backfill, ingest, and every gateway route run, and
fails on any request without exactly that header.

## What's stored

One SQLite-backed Durable Object (`Index`, binding `INDEX`, one global instance) holds a cache
of `dev.nibling.project`, `dev.nibling.feed.like`, `dev.nibling.graph.follow`, and
`dev.nibling.actor.profile` records: enough to answer `getRecent`, `getTrending`, and
`getKnown` without asking every PDS on every request. It is never the source of truth — compare
Tangled's Bobbin (D20) — and the schema lives in `src/index/model.ts`.

Not stored as a source of truth: project files (the author's PDS), git (tangled knots), previews
(the viewer's own browser), backlink counts such as "who liked this" (microcosm's Constellation),
and identity (PLC and DNS). The gateway caches reads of them for a while, and nothing more.

## How it rebuilds

Every 60 seconds, a Durable Object alarm (`Index.alarm`, `src/worker.ts`) runs one ingestion pass
(`ingest()`, `src/index/ingest.ts`):

1. If there's no stored cursor, or it's older than Jetstream's replay window (treated as 24h),
   backfill first: `com.atproto.sync.listReposByCollection` on the relay for each of the four
   collections, then `com.atproto.repo.listRecords` against each repo's own PDS
   (`src/index/backfill.ts`). A repo whose PDS can't be resolved or reached is skipped, not
   fatal. A repo that was listed completely has its other rows in that collection deleted, which
   catches deletions made while the index wasn't listening. The time the backfill started
   becomes the cursor, so the live tail replays everything that happened during it.
2. Open a live-tail WebSocket to Jetstream from the cursor and drain it until an event's `time`
   is within 5 seconds of now, or a 20-second budget runs out, applying events in order and
   saving the cursor after each one. The cursor is a Jetstream `time_us`. A drain that stayed
   open and heard nothing for its last 5 seconds moves the cursor up to when the socket opened
   (minus 5 seconds), but never more than 10 minutes past where it started: Jetstream sends
   nothing on a quiet filtered stream, and its replay scans about 12 minutes of network time per
   second, so a silent drain has only proven that much.
3. The alarm reschedules itself for 60 seconds later, in a `finally`, so a pass that throws still
   re-arms (`src/alarm.ts`). No outgoing WebSocket is held open between passes, so the Durable
   Object isn't billed for an idle connection.

The first alarm is armed from the Durable Object's constructor (`blockConcurrencyWhile`), so the
first request that wakes the object starts ingestion.

Relays, PDSes, PLC, and Slingshot are untrusted. Every call to them has a 10-second timeout
(`src/net.ts`). Listings stop on an empty or repeated cursor, and after 1000 relay pages or 100
PDS pages per repo and collection. A repo that answers with an error or a malformed page is
skipped, and its rows are not pruned. A PDS endpoint must be a public `https:` URL
(`isPublicHttpsUrl`), and `did:web` documents are fetched from the path the DID names
(`src/identity.ts`). Backfilled records must name the repo and collection that were listed.

Every record is validated against its generated `@nibling/lexicons` schema at the boundary
(`toPutEvent` in `src/index/jetstream.ts`, shared by the live tail and the backfill path); a
record that fails validation is dropped, not indexed. `IndexModel.apply()` is idempotent — in
Jetstream's words, delivery is at-least-once, so replaying the same create or the same delete
twice leaves the index unchanged — because the whole index can be thrown away and rebuilt from
the network at any time.

### Jetstream v2

Jetstream's live API is `wss://<host>/xrpc/network.bsky.jetstream.subscribeEvents`, with a
repeated `collections` param (an `app.bsky.feed.*`-style NSID-prefix wildcard works, so
`dev.nibling.*` does too), `kinds`, and a `cursor` that takes either a `seq` or a `time_us`, told
apart by magnitude. The index stores a `time_us`, since that also gives the cursor's age.
`src/index/jetstream.ts` targets this v2 shape; `wrangler.jsonc`'s `JETSTREAM_URL` points at
`wss://jetstream.us-east.bsky.network` accordingly.

`com.atproto.sync.listReposByCollection` takes `collection`, `limit`, `cursor` and returns
`{ repos: [{did}], cursor }`.

## Local dev

```sh
pnpm --filter @nibling/api test        # node --test, tests/**/*.test.ts
pnpm --filter @nibling/api typecheck   # tsc --noEmit
pnpm --filter @nibling/api dev         # wrangler dev (local only; CI deploys this worker)
```

The model tests run the same `Sql` interface the Durable Object uses, but over `node:sqlite`
(`DatabaseSync`) instead of `ctx.storage.sql`, so they need no Workers runtime. The router,
gateway, cards, jetstream-parsing, backfill, and ingest-loop tests all take their external
dependencies (the index stub, `fetch`, the WebSocket, `resolvePds`, the cache, the clock) as
injected parameters, so they also run under plain Node with no network and no Workers runtime. The
gateway tests use `memoryCache()`, which expires entries on the injected clock as the Cache API
does.

The Jetstream parser and ingest tests run on frames captured from the live v2 endpoint
(`tests/fixtures/jetstream/`): commit create and delete, identity, and account. Since nobody
writes `dev.nibling.*` yet, the `dev.nibling.*` fixtures keep a captured envelope and change only
`collection` and `record`.

### Reading through a local gateway

```sh
cd packages/api
npx wrangler dev --local --port 8787 --ip 127.0.0.1
```

`--local` keeps the Durable Object and the cache on this machine. The gateway still reads the real
network: PDSes, PLC, Slingshot, Constellation, and knots. Then point the playground at it:

```sh
VITE_NIBLING_GATEWAY=http://127.0.0.1:8787 pnpm --filter @nibling/app dev
```

With `VITE_NIBLING_GATEWAY` unset, the playground reads every service directly. Check the cache
from the command line: the second of two identical reads says `x-nib-cache: hit`.

```sh
curl -sD - -o /dev/null 'http://127.0.0.1:8787/xrpc/blue.microcosm.identity.resolveMiniDoc?identifier=natemoo.re' | grep x-nib-cache
```

This worker is never deployed from a development session. `.tangled/workflows/deploy.yml` deploys it to
`api.nibling.dev` on every push to `main`, with `pnpm run deploy`, which sets `APP_ORIGIN` to
`https://nibling.dev`.