# Gotchas

Accepted limitations and known traps: things that look like bugs but are
deliberate, and behavior that is easy to misread. Modelled on upstream Jetstream
V2's `specs/gotchas.md`.

If something here starts costing more than it saves, it belongs in
`docs/semantic-parity.md` as a real gap instead.

## Deliberate behavior that looks wrong

**`--backfill-only` reaches the sealed frontier, not the archive tip.** The
segment-download API serves sealed segments; the active segment is not part of
it. A backfill client stopping short of the tip is correct — an ordinary
subscriber replays sealed plus active durable rows and reaches everything. The
archive contract asserts both sides of that boundary on purpose.

**`--max-backfill-repos` caps dispatch per run, not the corpus.** A restart
under that flag picks up a *fresh* slice rather than replaying, because
completed repositories are skipped and the cap counts only dispatched ones.

**A `skipped` suite blocks admission exactly like a failure.** `scripts/admit`
records a suite that did not run rather than omitting it, and `verify` refuses
the digest. A partial run must never read as a full one.

**The lookback floor is computed over sealed segments only.** With every
fixture timestamp in the past, the floor clamps to the freshest *sealed*
segment's `min_seq`. Startup resumes the active segment in place rather than
sealing it, so the floor is not the active segment's first seq.

**Progress metrics are process-local; durable counts come from the store.**
`jetstream_backfill_completed_total` counts work done by *this* process. To ask
what is durably complete, read the status page's host aggregates. Confusing the
two makes a restart look like it redid everything.

## Traps in the harness, not the system

**A killed power-loss run leaves its NBD device claimed.** The next run dies
with `nbd: nbdN already in use` while `/sys/block/nbdN` still reports `size=0`
and no pid, which reads like a Stream failure and is not. Point
`STREAM_POWERLOSS_DEVICE` at another device (`nbd0`..`nbd3`). Repeated cycles
can wedge the Docker VM outright; restarting it is the repair.

**The simulator grows for as long as it is up.** It accumulates world state
continuously — multiple GB over a long session — and fixtures assume a small
world. A large one makes bootstrap outlast fixed timeouts in the harness;
a very small one makes bootstrap finish before a transient gauge can be
sampled. Reset it before a long run (`rm -rf` its data dir, restart), and note
that `scripts/admit` runs the two simulator-backed oracles *first* for this
reason.

**Do not sample a transient to prove a phase happened.** Whether
`jetstream_orchestrator_phase` is observable at all depends on how long
bootstrap takes, which depends on simulator size. Assert the cumulative
transition counter instead; it cannot be missed by arriving late.

**One injected fault can surface under two names.** The `relay/cursor` store
fault races to the process boundary either directly as `InjectedStoreFault` or
wrapped by the archive append that triggered it as `ArchiveAppendFailed`. Both
are correct and both exit non-zero. Pin the invariant — a fatal cause reaches
`main` — not one side of the race.

**A recycled Hetzner IP fails SSH as a host-key attack warning.** Tear down an
experiment and the next one can be handed an address a previous run used —
`65.109.0.222` came back as a build host having been experiment 3's watchdog,
and SSH refused with REMOTE HOST IDENTIFICATION HAS CHANGED. It is ordinary
address reuse, not an interception.

This is not just a manual-SSH nuisance: it broke a real provision. The watchdog
landed on a build host's recycled address, `arm_watchdog.sh` could not connect,
and the failed-apply trap tore the run down. `StrictHostKeyChecking=accept-new`
admits an *unknown* host but hard-fails a *changed* one, so it does not help
here. `provision.sh` and `arm_watchdog.sh` now scrub the host keys of the
servers they just created, which is safe precisely because those servers are
seconds old.

**Build the deploy artifact natively.** Emulating linux/amd64 on an arm64
workstation turns an 8-minute compile into 40-plus and yields a cross-emulated
approximation of the thing that ships. `STREAM_BUILD_HOST` points `admit` at a
native amd64 daemon; the image is streamed back so the push uses local
credentials and the build host never receives them.

**Repository size is heavy-tailed and `listRepos` is ordered largest-first.**
Median 10 KB, mean 0.45 MB, p99 10 MB, max 69 MB (400 independent random
cursors). The first 100,000 `listRepos` entries average ~10 MB — roughly 21x
the network mean. Two consequences:

- **Never sample consecutive entries.** They share a PDS and a creation cohort,
  so forty consecutive repos is one sample, not forty. Use independent random
  cursors.
- **Never extrapolate early crawl throughput.** The first hour is the heaviest
  0.5% of the network. Repos/s rises by more than an order of magnitude after
  the head; bytes/s is the more reliable health signal.

**The pinned simulator's repositories are ~4 KiB; real ones vary by 4 orders of magnitude.**
Measured against bsky.network: 15, 3.4, 2.2, 15, 4.6 and 37 MB in one sample.
Any failure that depends on real repository sizes — memory pressure, worker
contention, bandwidth saturation — is structurally unreachable offline. The
batch submit/consume deadlock survived 20 passing suites for this reason.

Reproducing that class costs about €0.03: a small cloud box crawling
bsky.network showed the deadlock in 90 seconds. Stream the image over SSH
(`docker image save | ssh 'docker image load'`) since the registry needs auth,
and write `/etc/docker/daemon.json` with explicit DNS first or BuildKit cannot
resolve. Reach for this whenever a hypothesis involves anything the fixture
cannot express; do not conclude from a green suite that the condition does not
occur.

## Open defects

**FIXED 2026-07-31 (stream `d9c4f90`, zat 0.3.23). `Accept-Encoding: identity`
on the getRepo hot path.** `fetched_car.zig` overrode the request encoding to
`identity`, so every `getRepo` call pulled uncompressed CARs. Kept here because
the fix itself has a trap.
Measured 2026-07-30 against real PDSes: they honour `gzip` and return **2.5x
smaller** bodies (they do *not* honour `zstd` — it silently falls back to
identity).

**Upstream does not do this.** It sets no `Accept-Encoding` and never sets
`DisableCompression`, so Go's `net/http` transparently requests and decodes
gzip, giving it 2.5x compression on this path.

**The override paired with a decoding constraint.** `Content-Encoding` is
transport-level, so the *decoded* body is byte-identical and verification is
unaffected — but Zig's `response.reader()` returns the **raw** body and does not
decompress. Only `response.readerDecompressing()` does. The override was
therefore correct *as paired with that reader*: deleting it alone would feed gzip
bytes straight into the CAR parser and break every fetch.

The real fix is both together — accept compression *and* switch to
`readerDecompressing` with a `std.compress.flate.max_window_len` buffer. Both
body-reading sites need it: the CAR stream, and `classifyErrorResponse`, which
matches on the error body's **text** and would silently fail every pattern
against a gzipped body, misclassifying a known error as unknown.

The override entered in `c52765b`.

**Throughput impact is small.** Measured at concurrency 100, gzip gave 136.7
repos/s against ~120 for identity — about 1.14x — because neither the test nor
production is bandwidth-bound (production runs 41 MB/s against 150+ available).
The main effect of the fix is pulling 2.5x less data from other operators'
PDSes across millions of requests. During that measurement a third run drew
4,337 errors against ~160 in the first two, consistent with throttling.

**The fix's own trap: the transfer buffer, not just the decompress buffer.**
`readerDecompressing(transfer_buffer, &decompress, decompress_buffer)` takes
*two* buffers. The decompress buffer needs
`std.compress.flate.max_window_len`; the **transfer** buffer feeds the
decompressor its compressed input, and a zero-length one does not return an
error — it **aborts the process** on the first gzip response. The first version
of this fix (`c8c8c5d`) passed `&.{}` there and would have aborted on the first
PDS that honoured the request. Caught by a loopback gzip test in zat, which
failed the same way; both sites now pass an 8 KiB transfer buffer.

zat had the mirror-image bug: it passed a real transfer buffer and a
*zero-length decompress* buffer, so gzip never worked there; the
`// disable gzip - zig stdlib issue` comment above it attributed the failure to
the stdlib, but the cause was the empty buffer. Fixed in 0.3.23, which also
bounds stalled transfers and replays connections closed before they answer.

**The status page divides by the wrong denominator.** `/status` reports
repository progress as `complete / total`, where `total` is the *enumeration*
count — how far `listRepos` has walked — not the network size. On experiment 6
at hour 57 it read:

```
repositories
  total          3,300,002
  complete       3,142,519 (95%)
```

The real network is ~5.7 M repositories (indigo's `account_repo`), so actual
coverage was ~58%. The enumeration total grows by 100,000 every batch and never
settles, so this percentage climbs toward 100% for the whole run regardless of
progress. Report the raw counts, or divide by a denominator that does not move.

**`StreamProcessRestarted` has never fired, across 35 restarts.** The rule is:

```
changes(process_start_time_seconds{job="stream"}[5m]) > 0
```

The batch-boundary restart takes the process down for ~4.5 minutes of startup,
during which the scrape target is gone and `process_start_time_seconds` has no
samples at all. It reappears with a new value, so within any 5-minute window
`changes()` typically sees a single sample and evaluates to 0. The rule cannot
observe a restart that outlasts its own window.

What did fire is `StreamTargetDown` (`up{job="stream"} == 0`), at 8 sample
points in 24 h. Restarts are visible in alerting, but through that rule and
only partially. Widening the `changes()` window past the startup duration, or keying
off `resets()` on a counter that survives, would make the intended rule work.

Fixed 2026-08-22 in `deploy/prometheus/rules.yml` (the in-repo source for the
rules; the box copy must be refreshed by hand). The rule is now

```
process_start_time_seconds{job="stream"} > time() - 900
```

which is true from the first post-startup scrape for 15 minutes and then
clears — one firing per restart regardless of scrape interval or startup
length. Since a single batch-boundary restart is by design, the same file adds
`StreamCrashLoop` (`changes(process_start_time_seconds[1h]) >= 2`) and
`StreamLiveTailStalled` (`time() -
jetstream_livestream_last_seen_upstream_event_timestamp_seconds > 600`,
guarded by `> 0`), the latter because `up == 1` never distinguished a wedged
live tail from a healthy one. None of these has a notification route yet.

**`HttpConnectionClosing` escapes to `main` and ends the process.** First
observed twice in experiment 6 (2026-07-28 09:56 and 13:24 UTC, roughly every
3.5 h), each exit immediately preceded by:

```
{"level":"ERROR","msg":"HttpConnectionClosing","component":"main"}
```

`component: "main"` is `logging.zig` renaming Zig's *default* log scope, which
is what the runtime uses when `pub fn main` returns an error — so the error
escaped every handler and unwound out of `main`. A remote closing an HTTP
connection is ordinary and must never be process-fatal; that is the same
invariant as the `setsockopt` fix, one layer up.

`ingest.zig` already carries a regression test for this class ("abrupt upstream
disconnects reconnect instead of ending the consumer"), written after an
earlier production exit, and it passes. So the *live consumer loop* is covered
and something else is not — most likely the bootstrap-phase HTTP client
(`getRepo`/`listRepos`/DID resolution), which the test does not exercise.
Finding the uncovered call site is the work.

Impact is bounded because durability holds: each restart resumed and the
durable counter never went backwards. Cost is ~60 s of crawl plus a firehose
reconnect per occurrence, about 17 over a 60-hour run.

**The interval is not stable — it accelerates as the crawl progresses.** By
hour 51 of experiment 6 the gap between process starts had shrunk roughly 4x,
and monotonically rather than noisily (30 starts retained in the logs):

```
gaps (h): 4.3 3.5 3.3 3.5 3.6 3.5 2.9 ... 1.5 1.4 1.2 1.1 1.0 0.9 0.9 0.9 1.0 1.3 1.1
first half median 2.89 h  ->  second half median 1.09 h
```

So "every 3.5 h" is the *early* rate, and quoting it understates the defect on
a long run. If you are estimating exposure, measure the recent gaps rather than
reusing this number:

```
docker compose logs --no-log-prefix -t stream | grep "subscribe controls:" | cut -c1-19
```

(`subscribe controls:` is the first line of a fresh process, so counting it
counts *process starts* — see the `RestartCount` caveat below for why that is
not the same as Docker's restart count.)

**FIXED (2026-07-31, not yet deployed): it was a keep-alive idle race on
`list_client`.** `engine.zig` creates one long-lived `zat.XrpcClient` for
`listRepos` and zat's transport defaults to `keep_alive = true`. A batch takes
~40 minutes to drain, during which `listRepos` is never called, so the pooled
connection sits idle until the relay closes it. The next page reuses that dead
socket, `receiveHead` reads zero bytes, and `HttpConnectionClosing` unwinds to
`main`. That is why the interval equals the batch-drain time *exactly*: the
connection is idle for precisely one batch.

`ade3d5d` fixed the identical race on the getRepo path by disabling keep-alive;
`list_client` never got the same treatment. `listReposPage` now retries up to
four times on a connection that died before answering — safe by construction,
since zero bytes received means nothing was applied.

**The restart is not what advances the dispatch loop.** The dispatch loop is
`paging: while (true)` and walks `listRepos` continuously in one process;
progress comes from the durable cursor, not from the restart. The restart was
crash recovery.

**The interval is not a timer at all — it is one failure per batch boundary.**
Experiment 6 ran `--backfill-batch-size=100000`. Discovery admits 100,000 repos,
the crawl drains them, and the process dies at the boundary where the next
`listRepos` page is fetched. So the interval is simply *batch size ÷ crawl
throughput*:

| throughput | 100,000 ÷ rate | matches observed |
|---|---|---|
| ~8 repos/s (early) | 3.5 h | the original "every 3.5 h" |
| ~27 repos/s (hour 51) | ~62 min | the "accelerated" ~1 h |

Nothing is degrading. The crawl got ~3x faster, so it hits the same
batch-boundary bug ~3x more often. The shrinking interval is not progressive
deterioration.

This also means the restart is load-bearing by accident: each fresh process
re-runs discovery and admits the next 100,000, so the run advances *because*
Docker restarts it. `jetstream_backfill_discovered_total` reads exactly `100000`
on a long-lived process for this reason — it is per-process and capped, not a
network total.

**Why this matters beyond lost crawl time.** The bounded-impact argument above
was measured during bootstrap, where progress is per-repo and restart-safe. A
death *during merge* is also covered, and that is verified rather than assumed:
`merge.zig` fsyncs dst rows before advancing the cursor, `commitMergeSource`
writes the cursor and rev updates in one `commitDurable` batch, and
`tests/powerloss_oracle.py` kills the process at
`after-merge-dst-flush-before-source-commit` on a real NBD device, asserting no
duplicate recovered events. Power loss is strictly harsher than a clean exit.

**The fix.** `HttpConnectionClosing` is raised by `std.http.Reader.receiveHead`
when a peer sends zero bytes of headers before closing — std's own
`Build/WebServer.zig` treats it as `=> return`, i.e. an ordinary event. Both
`http.Client` and `http.Server` include it via `http.Reader.HeadError`, so the
error type alone does not tell you which side hung up; find the call site rather
than inferring it. Zero bytes received also makes the request unambiguously
safe to retry — nothing was answered — so the correct handling at a client call
site is to retry once on a fresh connection instead of propagating. Any bare
`try ....receiveHead(...)` on a pooled keep-alive connection is exposed to this
race by construction.

**Measuring it: do not read `docker inspect ... ExitCode` on a *running*
container.** It reports 0 regardless, which reads as a clean exit and is simply
the absence of an exit. Use the process-start timestamps in the log, or
`RestartCount`, and note that those two count different things — Docker's
policy restarts versus actual process starts.

## Divergences that were never decided

**Host parking is no longer persisted.** Persistence entered in a commit whose
message does not mention parking. Upstream keeps parking in the runner's
memory, and the persisted version lacked upstream's rule that a park never
shortens. When checking whether a divergence was a decision, `git log -S` on
the identifier usually settles it.

## Dependency lessons

**websocket.zig's client set socket timeouts through `std.posix.setsockopt`,
which maps `BADF`/`NOTSOCK`/`INVAL` to `unreachable`.** Those are reachable on
a *connection* socket — the peer resets, or another thread closes the fd — and
`unreachable` cannot be caught, so an upstream dropping the connection killed
the process from inside the handshake. Fixed in the fork; the server side had
already taken the same fix. Any dependency that treats a syscall's
race-condition errno as impossible is a process-kill waiting to happen.

**The pins move together.** `zat` pins `websocket.zig` too, so bumping Stream's
websocket pin alone yields two module copies and
`file exists in modules 'build' and 'build0'`. Order: websocket → zat → Stream.

**"state live" on /status does not mean fresh content is flowing.** Check
seq advance across a window and median create-rkey TID age — never
`time_us`, and never a single sample.

**relay-eval under-reads during catch-up bursts**, and every deploy costs
one near-0% eval window (the measurement overlaps the startup storage
gate). A dedicated probe from the prod box is the tiebreak; the eval's
API is documented at relay-eval.waow.tech/llms.txt.

**Compaction passes cause 1–2s live-delivery lulls** (measured 2026-08-17:
ten >1s gaps in 10 minutes mid-pass, max 2.08s; zero above 0.4s while the
compactor idles). Consumers with strict sub-5s read timeouts will
misclassify these as dead connections — the relay-eval collector did
exactly that until its 2026-08-17 fix.

**`docker logs` vanish on container recreate**; Prometheus on the box
survives restarts and is the durable record.

**tangled short-sha links need `/commits/<sha>`** — the 404 page returns
HTTP 200, so a broken link looks alive to checkers.

**The static site deploys separately from the binary** (`just site`); a
green gate does not update index/stats/llms.txt.

**The edge rate limiter counts your own probes.** /metrics, /status, and
llms.txt checks from one workstation share that IP's anonymous 240/min
bucket — a verification loop can 429 itself and read as an outage.
Health checks from the box itself (localhost:6060) bypass caddy
entirely; prefer those in scripts.

**Caddy container recreate ≠ caddy reload.** Recreate (image change)
severs every proxied websocket once; reload (Caddyfile change) severs
nothing (`stream_close_delay`). Don't pay the recreate blip for a
config-only change.

**`listSegments` lags a compaction rewrite by up to minutes.** The rewrite
renames the new file over the old one first and refreshes the manifest after
(`compact/pass.zig`, `on_rewritten`), so a reader can fetch bytes whose header
checksum is newer than the checksum `listSegments` (and the `ETag`) advertise.
On a live-era segment with many DIDs the gap between the tmp file's last write
and the rename was ~6 minutes (segment 6720, 2026-08-29). Consumers that verify
checksums should treat a mismatch as "moving", skip, and re-list later —
strata's flow does exactly that — rather than retry in a tight loop.