atproto-crates
Fork of atproto-crates for SCN forked from ngerakines.me/atproto-crates
atproto pds permissioned-data rust
Star
Fork
Watch
atproto-crates tools conformance
1 folder · 6 files
lib
.gitignore
ci.sh
package.json
pnpm-lock.yaml
README.md
run.mjs
README.md

Reference-client conformance harness #

Boots a real atproto-pds and drives the real @atproto/oauth-client-node through PAR → authorize → token → resource request → refresh against it, and then through signOut(), checking that the grant ended with it.

tools/conformance/ci.sh          # build, install, run — exits non-zero on failure

That is the whole command, and it is run from anywhere — it resolves its own paths. It builds pds, installs the pinned client, and runs the checks. --runs=N changes how many sign-ins the soak performs (default 20). To run against a binary built somewhere else — another commit, say — use node run.mjs --pds-bin=/path/to/pds from this directory, after pnpm install.

If you have a CARGO_TARGET_DIR #

run.mjs looks for the binary in <repo>/target/debug/pds and <repo>/target/release/pds, and nowhere else. So a CARGO_TARGET_DIR in your environment, or a build.target-dir in a .cargo/config.toml, puts the build somewhere the search will not look, and a bare node run.mjs fails with:

Error: no pds binary found. Run:
  cargo build -p atproto-pds --bin pds --features clap
or pass --pds-bin=<path>.

after a build that succeeded. That message is telling the truth about the fix and lying about the cause — nothing is missing, it is in the other directory.

ci.sh handles this: it asks cargo metadata for the real target_directory and exports CONFORMANCE_PDS_BIN. Driving run.mjs yourself, pass the path:

node run.mjs --pds-bin="$CARGO_TARGET_DIR/debug/pds"
# or, equivalently
CONFORMANCE_PDS_BIN="$CARGO_TARGET_DIR/debug/pds" node run.mjs

Why this exists #

Two OAuth bugs shipped from this workspace and survived every test in it:

  • d7da6c7 — the DPoP nonce challenge on /oauth/par and /oauth/token was answered with RFC 9449 §9's resource-server form (401 + WWW-Authenticate) instead of §8's authorization-server form (400 + {"error":"use_dpop_nonce"}). @atproto/oauth-client builds its authorization-server fetch with isAuthServer: true and then recognises only the §8 shape, so it never retried and sign-in failed at its first request.
  • aaa5fa9 — SignaturePolicy::AnyS was a no-op on secp256k1, because k256 enforces low-S inside its own verifier. @atproto/jwk signs ES256K and does not normalise s, so roughly half of a conforming client's DPoP proofs were refused as "invalid signature".

Neither is subtle. Both were invisible from Rust, because the only client that ever spoke to this server was the Rust one: our DPoP client accepts either status and consults WWW-Authenticate first, and our signer always emitted low-S for a verifier that demanded it. Client and server agreed with each other and every test passed. The defects were visible only from TypeScript.

So the harness contributes exactly one thing: a client we did not write.

What it checks #

  1. Nonce challenge shape (RFC 9449 §8). A proof carrying no nonce is sent to /oauth/par and to /oauth/token; each must answer 400 with {"error":"use_dpop_nonce"} and a DPoP-Nonce header, and the retry carrying that nonce must get past the nonce check. Both endpoints, because fixing /oauth/par alone moves the failure to client.callback().
  2. A high-S ES256K proof is a valid proof. Two proofs are minted from one key over one request: (r, s) and (r, n−s). Both are valid ECDSA signatures over the same message under the same key and RFC 7515 blesses neither, so both must be accepted. The low-S request is the control — if it fails, the harness is wrong about how to build a proof and the high-S result cannot be read.
  3. Twenty cold sign-ins with the reference client. Each run constructs a fresh NodeOAuthClient — an empty nonce cache — and completes PAR, authorize, the code exchange, an authenticated getSession, and a forced refresh, asserting the refresh returned a different access token.

Cold is not a detail. fetchDpop stores the nonce off a response it did not understand as a challenge, so a warmed client succeeds on its second attempt and a broken challenge reads as flakiness rather than as a bug. And twenty runs is not superstition: the PDS advertises ES256K, compareAlgos prefers it unconditionally, so every run mints a secp256k1 DPoP key and signs several proofs with it. One run has an even chance of never producing the high-S form. Twenty do not.

The resource-server half of the nonce protocol (§9: 401 + WWW-Authenticate) is covered by check 3 rather than by a probe of its own. The two nonce spaces have separate HMAC domain separators, so the nonce the client cached during PAR is refused at /xrpc/, and the challenge has to be understood a second time — in the other shape. A server that answered both spaces the same way fails one check or the other whichever way it chose.

Proof that it detects the bugs it exists to prevent #

A harness that cannot catch its two founding bugs is decoration. Both were re-introduced by building pds from the commit before each fix and pointing the harness at it with --pds-bin:

build nonce shape high-S 20 cold sign-ins
d7da6c7^ — neither fix FAIL (/oauth/par answered 401) FAIL (high-S refused, low-S accepted) FAIL 20/20
aaa5fa9^ (= d7da6c7) — nonce fixed, high-S not ok FAIL FAIL 19/20
HEAD ok ok ok 20/20

The middle row is the one to look at. Nineteen of twenty sign-ins failed and one passed — that single lucky run is what a one-shot harness would have reported as a green build.

To reproduce. The three commands run in three different directories, which is the thing this recipe is easiest to get wrong — git clone . clones whatever repository you are standing in, so starting it from tools/conformance/ clones the right repo but leaves every relative path after it pointing one level too deep:

# 1. at the repository root — `.` is the repo being cloned
cd "$(git rev-parse --show-toplevel)"
git clone . /tmp/ablation
git -C /tmp/ablation checkout aaa5fa9^

# 2. in the clone — and with CARGO_TARGET_DIR explicitly cleared, see below
(cd /tmp/ablation && env -u CARGO_TARGET_DIR \
  cargo build -p atproto-pds --bin pds --features clap)

# 3. in tools/conformance/, where run.mjs and node_modules live
cd tools/conformance
node run.mjs --runs=20 --pds-bin=/tmp/ablation/target/debug/pds

Give the clone its own target/, which is what env -u CARGO_TARGET_DIR above is for. Sharing one does not work: two checkouts of the same workspace hash to the same -C metadata, so the clone's libatproto_identity.rlib overwrites yours and your next cargo build links the old one without recompiling it. The harness catches that — an atproto-pds carrying the nonce fix and an atproto-identity without the high-S one reports exactly the middle row above — but you will spend an hour deciding whether the bug is back.

Note that step 3 passes --pds-bin pointing into the clone's target/. That is the whole point of the flag here, and it is the same flag the CARGO_TARGET_DIR note at the top of this file is about.

How it is wired #

run.mjs ── lib/plc.mjs      a did:plc directory for one server, in memory
        ├─ lib/pds.mjs      spawns the real `pds` binary on a loopback port
        ├─ lib/gateway.mjs  the reverse proxy the PDS expects to sit behind
        ├─ lib/proofs.mjs   DPoP proofs built by hand, for checks 1 and 2
        └─ lib/flow.mjs     the reference client, for check 3

Nothing links against the crates. The harness talks to the pds binary over HTTP, so it can be pointed at a build of any commit.

Three pieces of scaffolding stand in for things a local server does not have, and each is worth understanding before trusting a result:

The gateway. issuer_url() in crates/atproto-pds/src/oauth/metadata.rs derives the OAuth issuer, the resource of the protected-resource document and the htu a pushed or token request is checked against from the service DID, always over https. The reference client refuses an https: URL whose host is loopback (httpsUriSchema in @atproto/oauth-types). A PDS is therefore only reachable by a conformant client from behind a TLS terminator, so the harness gives the server the identity pds.conformance.test and maps that origin onto the loopback socket exactly as a terminating proxy would: the URL's scheme and authority are replaced, X-Forwarded-Proto and X-Forwarded-Host are added (the server runs with PDS_TRUSTED_PROXY_HOPS=1, which is what makes a resource request's htu come out as the public origin), and nothing else is touched. Status, headers and body are handed back verbatim. That rule is not negotiable: the 401-vs-400 bug was diagnosed with a shim that rewrote a status code, and a gateway that edits a response is a gateway that can hide the next one.

The PLC directory. createAccount mints a DID by POSTing a signed genesis operation to the configured directory, and the client resolves that DID back to a document to check that the token it was handed came from the server the account actually lives on. lib/plc.mjs answers both, rendering the operation the PDS submitted as the document it describes. It does not verify signatures or order a log — it is a fixture, not an implementation.

The consent POST. client.authorize() returns a URL a human would open. The harness posts the credentials to /oauth/authorize — the JSON endpoint the consent form itself submits to — and carries code, state and iss back to client.callback(). That is the only step of the flow the harness performs rather than the client.

That shortcut has a cost, and it has already been paid once. Posting the JSON directly means the consent page itself is never exercised — not the form, not the inline script that lifts its submit into that JSON, not the redirect the script assembles afterwards. Every request the harness makes to /oauth/authorize is one the page would have made correctly. So a bug living in the page is invisible here in the same way the two founding bugs were invisible to the Rust suite, and for the same reason: the only client that ever drives the consent page is a browser, and no browser runs in this harness.

One did get out that way. The script called preventDefault, then .json() on every response, and attached nothing to the rejection — so any answer that was not a JSON body (an empty-bodied 408 from the request deadline, a 503 from load shedding, an HTML error page from a proxy in front of the origin) left the page doing nothing at all, for ever. It was reported to us as a PDS that hangs. The invariants are pinned in oauth::consent's unit tests now, which is a weaker thing than running them: those tests read the rendered document, they do not execute it. A harness that drove a real browser through the page would be the honest version, and does not exist yet.

The loopback client identity (http://localhost?redirect_uri=…&scope=atproto) is a spec-defined development client, so no client-metadata document has to be served. The redirect URI is never listened on; the harness intercepts the code before a browser would be redirected anywhere.

How CI runs it #

.tangled/workflows/ci.yml has run this harness since c668dd0. The step is oauth conformance, near the end of the file:

  - name: "oauth conformance"
    command: |
      ulimit -n "$(ulimit -Hn)" 2>/dev/null || true
      tools/conformance/ci.sh

It is last but for audit on purpose: it is the only step that boots a server and the only one that installs an npm tree. dependencies.nixpkgs carries nodejs and pnpm for it — pnpm specifically, so the install honours pnpm-lock.yaml; ci.sh falls back to npm, which ignores that lockfile and takes its versions from the exact pins in package.json instead. coreutils is there in part for this script's dirname.

What it costs, and what it touches #

The checks themselves take a few seconds once the binary is built — three checks, twenty sign-ins, several hundred requests — and reach nothing but loopback. The PLC directory, the client metadata and the handle resolution are all local fixtures.

ci.sh as a whole does reach the network, and it is worth being precise about where. pnpm install --frozen-lockfile fetches the pinned client from registry.npmjs.org. That install is the harness's only npm dependency, and in CI it is the only fetch this step makes: the step's cargo build finds the crates the Rust steps before it already downloaded. (From a cold Cargo cache, as on a fresh machine, the build fetches the locked crate graph from crates.io first, like any build.) So when this step fails in a way that smells like DNS or TLS rather than OAuth, the install is the first thing to rule out. Once the tree is on disk, nothing further leaves the machine.

It is not the workflow's only outbound dependency. rustup downloads the pinned toolchain in the toolchain step, submodules clones dasl-testing from GitHub, the first step that builds fetches the locked crate graph from crates.io (nothing is vendored or built --offline), and audit fetches the RustSec advisory database. A DNS- or TLS-shaped failure in one of those steps is that step's, not this harness's.

So "the network is never reached" is true of the checks and false of the script. A run whose pnpm install was served entirely from a warm pnpm store touches nothing at all, which is why the distinction is easy to lose locally and sharp in CI.