perf.yml (push to main, pull requests to main, manual) is the gate. ci.py runs the report tests, builds the CLI and a snapshot daemon, replays the recorded bonsai scenarios once (five fresh-home samples each), publishes the run, and exits with the gate status after publication. It never repeats a measurement. The replay result is printed as JSON in the pipeline log, so fork PRs without secrets still show it. A replay that crashed leaves a partial result with no gated numbers; ci.py publishes nothing then.
perf-live.yml (push to main, manual) runs the same scenarios once against a real model through live_ci.py and records the result. Live never gates. It is a separate workflow because a live run takes longer than the gate should (the long and kernel scenarios alone are about 20 minutes of real generation), and spindle stops a workflow at 40 minutes.

Pipeline secrets, under repository Settings -> Pipelines -> Secrets:

WISP_APP_PASSWORD: an app password for okami.mom, used to write records and deploy sites.
BONSAI_URL and BONSAI_TOKEN: optional, live runs against bonsai (preferred when both are set).
OPENROUTER_API_KEY: optional, live runs against OpenRouter when bonsai is not configured.

isolation #

The daemon, CLI and kernels run model output, so both workflows run run.py as the perf-agent user through sudo -n -u perf-agent env -i, passing only the variables in isolation.py, with writable directories under a 0700 directory perf-agent creates in /tmp. Before anything runs, Sandbox.open checks that the sudo rule exists, that perf-agent can read the checkout, and that perf-agent cannot read the CI processes' /proc/<pid>/environ or mem; any failure stops the job. All perf-agent processes are killed before a site is deployed, because wispctl takes its password in argv. The spindle guest gets the user and the sudo rule from ark ec75712.

Live holds the provider token in a loopback gateway inside live_ci.py (the CI uid). It forwards only GET /v1/models and POST /v1/chat/completions to one upstream base URL, replaces any Authorization header with the token, and refuses requests beyond PERF_LIVE_MAX_REQUESTS (250; the eight recorded scenarios make 190). When BONSAI_URL and BONSAI_TOKEN are both set, live uses bonsai: the model is the first id from its /models, there is no quota check, requests are paced at 25 per minute (ci.klbr.net rate-limits about 30), and the gateway forwards one chat at a time, because the route answers a second concurrent chat with 429. The route also forces bonsai-2-27b and clamps max_tokens to 4096. Bonsai answers 429 while another caller holds it; live waits for it, polling its model list every 30 seconds for up to 12 minutes, and records skipped: bonsai busy if it never frees. A 429 after the run has started stays a recorded failure. BONSAI_URL without BONSAI_TOKEN is logged and ignored. Otherwise live uses OpenRouter, only when /api/v1/key reports at least twice the request cap in free_model_daily_requests.remaining, with cohere/north-mini-code:free pinned and requests paced at 18 per minute. These constants live in perf-live.yml. A skipped or failed live run is recorded with its reason (event, unset key, quota, HTTP 429, model unavailable, gateway budget) and the upstream status counts; live is never retried.

where results live #

Nothing is ever redeployed. Each run writes atproto records in okami.mom's repo; records hold only strings and integers, because the atproto data model has no floats, so numbers travel as JSON text.

A replay run is a pet.nekomimi.albedo.perf.run record, rkey = the commit sha (<sha>-2 and so on when a commit runs again). summary holds the gated numbers and per-scenario memory and wire values; the full results.json is the result blob; committedAt orders main's history. The run also gets its own site, albedo-perf-<sha7> (same suffix as the rkey), at https://sites.wisp.place/okami.mom/albedo-perf-<sha7>/; CI logs it as perf report: <url>.
A live run is a pet.nekomimi.albedo.perf.live record, rkey = the commit sha. status says whether it ran and why not, scenarios holds what the run page draws, and the full live results.json is the result blob. It has no site: run pages load it at view time, so a live run recorded after the replay site was deployed still shows there.

publish.py replay uploads the blob, claims the record (createRecord with an explicit rkey fails if it exists, so two runs never share a record or a site), renders the site, deploys it, and reads data/run.json back. If the deploy fails, the record is deleted, so no record points at a missing site. publish.py live writes the live record. Both refuse a replay result without gated numbers.

The gate's baseline is the latest main-push run record whose commit is an ancestor of the commit under test (git merge-base --is-ancestor, after unshallowing the clone); ci.py fetches its result blob and passes it to run.py --baseline. With no such record, only budgets apply.

the run page #

Each run's page answers whether the commit made albedo heavier and where. A header bar holds the gate chip, a one-sentence verdict, the commit and a picker for the run to compare with (default: its baseline main run). Tiles show the headline numbers (six, plus kernel at rest, kernel memory per cell and the kernel pool when the result carries them; the per-turn tile is the BEAM leak gate with process memory as a secondary line) in decimal units with their change against the compared run, a budget bar, and a sparkline once main has three runs; exact bytes and noise are in tooltips. A number measured once is labelled as such and only counts as moved at 5% or more. Arrows are red only when the gate failed that metric. "What moved" lists changes larger than three times the noise. Each scenario card charts the private memory kept after each step with each turn's peak, and bytes per model request; its raw numbers are behind a disclosure. A Replay/Live toggle switches the scenario cards to this commit's newest live record. The history chart covers every main push, with metric tabs, a ±3×MAD noise band and the budget when it is near the data.

The page fetches records at view time from the PDS's public com.atproto.repo.listRecords (CORS *, paged 100 at a time), so an old page compares against runs made after it. Run records whose summary has no gated numbers are ignored. Charts use Apache ECharts 6.1.0 from jsDelivr, pinned with an SRI hash of dist/echarts.min.js taken from the npm tarball (verified against the registry's sha512), with the canvas renderer and the built-in default and dark themes switched at runtime with prefers-color-scheme. Without JavaScript the page shows the gate's numbers in a table.

local use #

With wispctl on PATH and WISP_APP_PASSWORD in the environment:

python3 test/perf/report/publish.py replay /path/to/replay-results.json
python3 test/perf/report/publish.py live status.json --result live.json --commit <sha> --ref <ref> --event manual

render.py SITE_DIRECTORY results.json renders one run's site offline. --did and --pds point its page at another repo, for example a local mock PDS serving listRecords.