core
Monorepo for the nibling.run platform
Star 0
Fork 0
Watch
core packages api
2 folders · 4 files
src
tests
package.json
README.md
tsconfig.json
wrangler.jsonc
README.md

@nibling/api #

The one stateless Cloudflare Worker nibling.run runs (D22). It is the app's read path: a gateway that answers repo reads, identity, backlinks, and git fetches from a cache. The app works without it, reading each service directly, with no global feed, no trending, no link cards, and nothing from run.nibling.actor.getKnown. All project data, likes, follows, and profiles live on each account's own PDS. This worker never holds that data as the source of truth.

Routes #

  • GET /xrpc/run.nibling.feed.getRecent?limit=&cursor=&tag= — recently created or updated projects, newest first. A project sorts by its updatedAt (or createdAt), clamped to when the index saw that version, so a far-future date can't pin it to the top. Like and fork counts count distinct accounts.
  • GET /xrpc/run.nibling.feed.getTrending?limit=&cursor= — projects ranked by likes and forks from the last 7 days, decayed linearly to zero at the window edge. A like or fork counts from its createdAt, but never later than when the index first saw it, so a post-dated record can't stay fresh.
  • GET /xrpc/run.nibling.actor.getKnown?dids= — which of the given DIDs have a run.nibling.project or run.nibling.actor.profile record. Used to narrow imported Bluesky or Tangled follows to accounts that already use nibling.
  • GET /:actor/:name — a project page: Open Graph tags for crawlers, then a <meta http-equiv="refresh"> redirect to the app for everyone else. :name is the project's slug or its rkey; a slug is found by listing the author's PDS. Links use the slug when there is one.
  • GET /:actor/:name/embed — the same metadata, but the page itself is an iframe-friendly shell around the app's /:actor/:name/embed (nothing navigates away).
  • GET /@:actor/:name and /@:actor/:name/embed — a 301 to the same path without the @.
  • GET /oembed?url=https://nibling.run/actor/name — oEmbed JSON (type: "rich") for the page above.
  • GET /oauth/client-metadata.json, the atproto OAuth client metadata document (D19), built by @nibling/sdk/metadata so the app and the worker share one scope string.

The XRPC routes send Access-Control-Allow-Origin: * and a short Cache-Control, and error as { "error": "...", "message": "..." }.

Gateway #

Every read the app makes goes through these routes (src/gateway/), all GET unless noted, all with Access-Control-Allow-Origin: *. OPTIONS on any path answers a preflight that allows content-type, git-protocol, and x-nib-fresh.

Route Upstream Cached for
com.atproto.repo.getRecord with cid the repo's PDS 1 year, and only when the answer carries that cid
com.atproto.repo.getRecord without cid, listRecords, describeRepo the repo's PDS 15 s, then served stale for 60 s while it refreshes
com.atproto.sync.getBlob the repo's PDS 1 year
blue.microcosm.identity.resolveMiniDoc Slingshot, then PLC or did:web and DNS-over-HTTPS 5 minutes
com.atproto.identity.resolveDid PLC or the did:web host 5 minutes
blue.microcosm.links.getBacklinks, getBacklinksCount Constellation 15 s
blue.microcosm.repo.getRecordByUri (at_uri, optional cid) Slingshot 15 s with the stale window, or 1 year with cid
/git/<repo DID>/info/refs?service=git-upload-pack the repo's #tangled_knot never
POST /git/<repo DID>/git-upload-pack the repo's #tangled_knot 1 year when the body is only want, deepen, and done lines, keyed by its SHA-256; otherwise never
  • Every response says how it was served in x-nib-cache: hit, stale, miss, or bypass. A request with x-nib-fresh: 1 skips the cached answer and stores the new one, which is how a writer reads their own write. The SDK sends it for the signed-in account's own repo.
  • The Cache API is per data center, and a purge does not reach the others, so nothing purges. Every entry ages out on its TTL. The Cache API ignores stale-while-revalidate, so src/gateway/cache.ts serves stale entries itself and refreshes them with waitUntil. Only a 200 is stored.
  • Each route checks its params at the boundary (a DID or handle through @nibling/sdk/identifier, an NSID, a record key, a CID, a limit from 1 to 100) and drops any other param. A repo handle is resolved and forwarded as its DID, so the handle and the DID share one cache entry.
  • The PDS or knot comes from cached identity and must be a public https: URL (isPublicHttpsUrl). Redirects are followed by hand, at most 3, and each hop is checked the same way. The git routes proxy only to the knot the repo DID's document names, never to a host from the request.
  • Upstream calls time out after 10 seconds (60 for git). A JSON answer or a blob over 5 MB, and a pack over 100 MB, is cut off. An upstream 5xx, timeout, or cut-off becomes a 502 or 504, which tells the SDK to retry on the direct path. A 4xx passes through as an answer.
  • Blobs are served with Content-Security-Policy: default-src 'none'; sandbox and X-Content-Type-Options: nosniff, since an uploader picks the content type.
  • git-receive-pack is refused with a 403. Pushes, like PDS writes, go from the browser straight to the knot or PDS: OAuth's DPoP tokens are bound to a browser key, and a push token is single-use and expires in 60 seconds.

User-Agent #

Every request this worker makes carries User-Agent: nibling.run/<version> (+https://nibling.run/contact), so a service can tell us about runaway traffic. Set CONTACT_URL to change the address. upstream() (src/upstream.ts) adds the header, and only it makes the branded Upstream type that the router, gateway, backfill, ingest, and identity code accept, so a bare fetch does not typecheck. Jetstream is opened with a fetch carrying Upgrade: websocket rather than new WebSocket, so its handshake carries the header too. src/services.ts is the only file that reads globalThis.fetch. tests/user-agent.test.ts replaces the global fetch while link cards, backfill, ingest, and every gateway route run, and fails on any request without exactly that header.

What's stored #

One SQLite-backed Durable Object (Index, binding INDEX, one global instance) holds a cache of run.nibling.project, run.nibling.feed.like, run.nibling.graph.follow, and run.nibling.actor.profile records: enough to answer getRecent, getTrending, and getKnown without asking every PDS on every request. It is never the source of truth — compare Tangled's Bobbin (D20) — and the schema lives in src/index/model.ts.

Not stored as a source of truth: project files (the author's PDS), git (tangled knots), previews (the viewer's own browser), backlink counts such as "who liked this" (microcosm's Constellation), and identity (PLC and DNS). The gateway caches reads of them for a while, and nothing more.

How it rebuilds #

Every 60 seconds, a Durable Object alarm (Index.alarm, src/worker.ts) runs one ingestion pass (ingest(), src/index/ingest.ts):

  1. If there's no stored cursor, or it's older than Jetstream's replay window (treated as 24h), backfill first: com.atproto.sync.listReposByCollection on the relay for each of the four collections, then com.atproto.repo.listRecords against each repo's own PDS (src/index/backfill.ts). A repo whose PDS can't be resolved or reached is skipped, not fatal. A repo that was listed completely has its other rows in that collection deleted, which catches deletions made while the index wasn't listening. The time the backfill started becomes the cursor, so the live tail replays everything that happened during it.
  2. Open a live-tail WebSocket to Jetstream from the cursor and drain it until an event's time is within 5 seconds of now, or a 20-second budget runs out, applying events in order and saving the cursor after each one. The cursor is a Jetstream time_us. A drain that stayed open and heard nothing for its last 5 seconds moves the cursor up to when the socket opened (minus 5 seconds), but never more than 10 minutes past where it started: Jetstream sends nothing on a quiet filtered stream, and its replay scans about 12 minutes of network time per second, so a silent drain has only proven that much.
  3. The alarm reschedules itself for 60 seconds later, in a finally, so a pass that throws still re-arms (src/alarm.ts). No outgoing WebSocket is held open between passes, so the Durable Object isn't billed for an idle connection.

The first alarm is armed from the Durable Object's constructor (blockConcurrencyWhile), so the first request that wakes the object starts ingestion.

Relays, PDSes, PLC, and Slingshot are untrusted. Every call to them has a 10-second timeout (src/net.ts). Listings stop on an empty or repeated cursor, and after 1000 relay pages or 100 PDS pages per repo and collection. A repo that answers with an error or a malformed page is skipped, and its rows are not pruned. A PDS endpoint must be a public https: URL (isPublicHttpsUrl), and did:web documents are fetched from the path the DID names (src/identity.ts). Backfilled records must name the repo and collection that were listed.

Every record is validated against its generated @nibling/lexicons schema at the boundary (toPutEvent in src/index/jetstream.ts, shared by the live tail and the backfill path); a record that fails validation is dropped, not indexed. IndexModel.apply() is idempotent — in Jetstream's words, delivery is at-least-once, so replaying the same create or the same delete twice leaves the index unchanged — because the whole index can be thrown away and rebuilt from the network at any time.

Jetstream v2 #

Jetstream's live API is wss://<host>/xrpc/network.bsky.jetstream.subscribeEvents, with a repeated collections param (an app.bsky.feed.*-style NSID-prefix wildcard works, so run.nibling.* does too), kinds, and a cursor that takes either a seq or a time_us, told apart by magnitude. The index stores a time_us, since that also gives the cursor's age. src/index/jetstream.ts targets this v2 shape; wrangler.jsonc's JETSTREAM_URL points at wss://jetstream.us-east.bsky.network accordingly.

com.atproto.sync.listReposByCollection takes collection, limit, cursor and returns { repos: [{did}], cursor }.

Local dev #

pnpm --filter @nibling/api test        # node --test, tests/**/*.test.ts
pnpm --filter @nibling/api typecheck   # tsc --noEmit
pnpm --filter @nibling/api dev         # wrangler dev (local only; this worker is never deployed from here)

The model tests run the same Sql interface the Durable Object uses, but over node:sqlite (DatabaseSync) instead of ctx.storage.sql, so they need no Workers runtime. The router, gateway, cards, jetstream-parsing, backfill, and ingest-loop tests all take their external dependencies (the index stub, fetch, the WebSocket, resolvePds, the cache, the clock) as injected parameters, so they also run under plain Node with no network and no Workers runtime. The gateway tests use memoryCache(), which expires entries on the injected clock as the Cache API does.

The Jetstream parser and ingest tests run on frames captured from the live v2 endpoint (tests/fixtures/jetstream/): commit create and delete, identity, and account. Since nobody writes run.nibling.* yet, the run.nibling.* fixtures keep a captured envelope and change only collection and record.

Reading through a local gateway #

cd packages/api
npx wrangler dev --local --port 8787 --ip 127.0.0.1

--local keeps the Durable Object and the cache on this machine. The gateway still reads the real network: PDSes, PLC, Slingshot, Constellation, and knots. Then point the playground at it:

VITE_NIBLING_GATEWAY=http://127.0.0.1:8787 pnpm --filter @nibling/app dev

With VITE_NIBLING_GATEWAY unset, the playground reads every service directly. Check the cache from the command line: the second of two identical reads says x-nib-cache: hit.

curl -sD - -o /dev/null 'http://127.0.0.1:8787/xrpc/blue.microcosm.identity.resolveMiniDoc?identifier=natemoo.re' | grep x-nib-cache

This worker is never deployed from a development session: no wrangler deploy, ever.