# Performance checks `test/perf/run.py` measures retained memory and HTTP payload bytes through the real daemon. Replay uses committed bonsai SSE responses and gates every commit. Live mode uses an OpenAI-compatible provider and never gates. ## Commands ```sh nix develop -c go -C cli build -o bin/albedo ./cmd/albedo nix develop -c python3 test/perf/run.py --mode replay --runs 5 --out /tmp/perf/results.json [--baseline main-results.json] python3 test/perf/compare.py /tmp/perf/results.json [--baseline main-results.json] ``` `ALBEDO_TEST_DAEMON` accepts a launcher from `test/snapshot-daemon.sh`; without it the shared E2E harness builds one snapshot for the invocation. `ALBEDO_TEST_CLI` selects a prebuilt CLI and `PERF_COMMIT` names the commit when the checkout has no usable git. Each repetition gets a fresh hermetic daemon home, and the harness stops every private daemon on success and failure. A failed measurement is not retried, and a scenario that stops early reports its own error rather than the recording's unused responses. In replay, a scenario's repetitions run at the same time, each on its own daemon (`--jobs 1` runs them in turn; live and recording runs are always sequential). On valefar at CI size (12 CPUs, 12 GiB) a five-run replay of all eight scenarios takes about 2.4 minutes of scenario time concurrently and 10.5 minutes sequentially. `--scenario NAME` selects a diagnostic subset and derives no gates. Scenarios: boot (empty daemon, then one idle session); chat (five text turns); coding (eight cells that read and edit a fixture and run its tests); long (60 text turns: 20 warm-up, then 40 gated, with HTTP and `albedo sessions read` hydration at the end); fan-out (one turn in each of 20 sessions); agents (a parent and three children that mail it); kernel (one cell at rest, five warm-up cells, then 30 cells of ordinary work); pool (a cold kernel, its release, a wake that warms spares, a second session adopting a forked spare, then rest). ## Memory Linux measurements need `/proc//smaps_rollup` and an inspect-enabled daemon. `measurement.py` reuses `test/manual/agent_density.py` process discovery, which on Linux identifies helpers by the role name they give themselves (`albedo-kernel`, `-spare`, `-zygote`, `-bridge`) and claims a reparented one by `ALBEDO_HOME` in its environment. One `erl` node per measured daemon stays connected to the inspect node and calls `test/perf/albedo_perf_probe.erl`; starting an `erl` per call cost about 0.5 s each. A retained checkpoint waits 300 ms for the session's two quiet hibernation ticks, runs a major GC of every BEAM process, then reports `erlang:memory()` by category, each process's PSS and USS, each session actor's memory with the processes it owns, and each held kernel's USS. A 50 ms sampler records the tree's peak PSS and USS and the BEAM total between checkpoints; a peak is a sampled maximum, not an allocation trace. Every turn is measured except in long and kernel, which take a checkpoint at the end of warm-up and on every fifth turn after it; wire bytes are counted on every turn. The tree gates use USS, not PSS. PSS divides shared file pages among every process that maps them, so another BEAM on the host moved the empty daemon's PSS by 8 MB; USS is private to the measured processes. PSS stays in each scenario's report. Daemon USS still moves in allocator-carrier steps of 2 to 5 MB between checkpoints, which BEAM allocated memory does not. Concurrency does not move the gated numbers: in three five-run batches, two concurrent and one sequential, every sequential median lay within the spread of the concurrent ones, with no larger MAD. ## HTTP bytes `transport.py` puts a transparent TCP proxy in front of the provider gateway and another in front of the daemon API. Counts include HTTP headers, request compression and chunk framing, and exclude TCP/IP headers. Provider rows carry request and response bytes and the reported prompt and cached tokens, attributed to the turn that caused them. A fixed full-state read after each settled turn measures client hydration; readiness polling bypasses the measured connection. Provider request bytes are identical across replay runs. Hydration bytes vary by up to about 160 B in 73 KB, because event payloads carry timestamps and durations; the one-percent byte rule absorbs this. The replay gateway serves the recorded status, headers and exact SSE body bytes with content-length framing; it does not reproduce upstream packet timing. Recordings were made at temperature zero, seed 42, max_tokens 512. ## Freshness Each recorded request has SHA-256 digests for its system prompt, tool schemas and message shapes. Per-run fixture paths and UUIDs are normalized before hashing. Replay reports changed categories in `scenarios..recordings.stale_turns`; staleness does not fail the memory or byte gate. Recordings retain their UTC creation time and model. A stale recording needs deliberate refresh against bonsai, not a synthetic response edit. ## Retained growth per turn Long's first 20 turns are reported as `warmup_retained` (memory after turn 20 minus before turn 1, about 3.4 MB of USS) and not gated. Over turns 20 to 60, `retained_turn_beam` is about 0 (medians 1.5, 4.4 and 1.9 KB per turn in three batches) and is the precise leak gate. `retained_turn_tree_uss`, process memory per turn, is 10 to 16 KB: the daemon's SQLite writer connection keeps SQLite's default 2 MB page cache (readers and the checkpoint connection set `cache_size=-128`), and a 160-turn diagnostic showed daemon memory still growing 12 to 13 KB per turn through turn 120 and 2.6 KB per turn over turns 120 to 160, matching that cache filling while the transcript grows about 2.7 KB per turn. The writer cache is a deliberate one-time cost, so this number has a 24 KB regression floor and a budget instead of a target of zero. ## Kernels `kernel_rest_uss` is one kernel's USS after its first cell (16.3 MB in every run). `kernel_cell_slope` is the kernel's USS per cell over cells 6 to 36 (about 8 KB). A 600-cell diagnostic on valefar, cycling the recorded cells, showed that growth is bounded: kernel USS climbs about 2 KB per cell, then drops by about 1.2 MB at irregular points (near cells 250 and 350), and after 600 cells sat 1.2 MB below cell 10. The gated 30-cell slope measures the first ramp of that sawtooth, so a per-cell leak adds to it and shows as a regression. The pool scenario runs the product's pool defaults (two spares, the Linux zygote) with its timers at `ALBEDO_TEST_TIME_SCALE=6`. `kernel_pool_pss` is what remains after every kernel is released and the spares cool down: 0, since the zygote exits after its idle minute. Reported but not gated: a cold-launched kernel costs 16.3 MB USS, a kernel forked from the zygote 6.6 MB, and the warm pool (zygote and spares) 19.9 MB PSS. Under a scaled clock the zygote's idle minute and the spare cool-down are scaled too; before that, an unscaled timer left the zygote alive at a scaled rest checkpoint. ## Comparison Results schema version 1 contains each scenario's medians, median absolute deviations (MAD), raw samples, request rows and recording freshness, plus the derived gated numbers and their verdicts. A single-sample gate carries `single_sample`. `budgets.toml` holds an absolute ceiling and a regression floor per gated number, set from the three batches above; its comments give the measurements behind each. A memory number regresses when it exceeds the baseline by more than the largest of three times the baseline's MAD, five percent of the baseline, and its floor. Only the baseline's noise counts: a regression that also adds noise must not widen its own margin (the deliberate leak raised per-turn BEAM MAD from 1.5 to 11 KB). A byte count regresses past one percent. A number over its ceiling fails with or without a baseline, so repeated small regressions cannot accumulate. With no ancestor result there is no baseline and only budgets apply. The gate catches both injected regressions on scratch branches at CI size against a batch as baseline: keeping a 40 KB copy of each submitted operation in the session actor raised `retained_turn_tree_uss` to 53.7 KB and `retained_turn_beam` to 23.6 KB per turn (both over budget, and regressed), and an unused 7.6 KB tool schema raised `request_bytes_per_turn` from 48,704 to 56,421 B (regressed and over budget). The other two batches compared against the same baseline pass. Live results are observations and never fail the gate. Live replies are slow enough that the 3 s model-wait release frees kernels between cells, so kernel numbers come from replay only. ## Output safety Private daemons receive an explicit environment allowlist: PATH, HOME and TMPDIR under the fixture, LANG, fixed VM flags, and the ALBEDO variables required for the private snapshot. They never inherit the caller's provider credentials. The daemon profile uses a dummy key. Only an external provider gateway may hold a real key. `SecretScanner` rejects provided credential values and recognizable API keys, AWS credentials, JWTs and private-key headers before a result or recording is written. It also scans decoded SSE and reconstructed model deltas, since a key can span chunks. Report generation must scan every generated or restored site file before publication. Errors report the affected output and match category, never a matched value. A secret scan is a second check, not a sandbox: live tools require an isolated execution boundary. MacOS supports recording orchestration only; it does not report Linux PSS/USS. Live model tools must not execute under the developer's host account. The recording launcher provides the required container boundary. CI live execution similarly requires a separate unprivileged UID from the key-holding gateway, or is skipped with a no-isolation reason. ## Isolated recording Run `python3 test/perf/record_container.py --out /tmp/perf-recordings-` on valefar from a checkout other than the working one. The launcher copies the tracked source to a private temporary directory (mode 0700), builds the snapshot daemon and CLI inside `nix develop`, and records every scenario in one container. The local `alpine:3.22` image must exist; the launcher pins it by image ID. `--memory 8g`, `--cpus 4`, `--pids 4096` (the fan-out scenario's 20 sessions need it) and `--timeout 3600` bound the run; `--scenario` is diagnostic only. The container runs as UID 65534 with every capability dropped, no-new-privileges, a read-only root, Docker's default seccomp profile, tmpfs `/home` and `/tmp`, and `--network none`. Source, daemon, CLI and the Nix-store closure of the commands they run are read-only mounts; `ps` comes from the flake's locked nixpkgs. The recordings directory is the only writable mount. No host environment, Docker socket, credentials or home directory enters it. Docker's `runsc` (gVisor) runtime is used when present; valefar has only runc. A host bridge forwards a Unix socket to bonsai at 127.0.0.1:8899 and nowhere else; inside, a second bridge serves that socket on 127.0.0.1:8899. Before recording, a real agent cell in the same container must fail to reach 1.1.1.1, 10.0.0.30 and the Docker host, fail to read an unmounted host file, and see no Docker socket; `isolation.json` keeps the result. `/tmp/bonsai-gpu.lock` is held only while the recording container runs, not during the build or the probe. Afterwards the container is removed, a second container with the same limits makes the output readable, `SecretScanner` checks every file, and only then are the files copied to `--out`. A failed run copies nothing. Commit the scenario recordings, not `results.json` or `isolation.json`.