# deploying — routine code changes to the live box

How to ship a commit to the running instance at stream.waow.tech, what each
stage costs, and what to expect while it happens. This is the everyday flow;
the whole-network *experiment* procedure (provisioning, watchdogs, cost
envelopes) is `deployment-runbook.md`.

The instance is one Hetzner box (stream-cx43, 89.167.122.160, hel1) running
docker compose at `/opt/stream-experiment/`. The stream image is pinned in
`/opt/stream-experiment/.env` as `STREAM_IMAGE=atcr.io/zat.dev/stream:<sha>`.

## where the gate runs (this repo is not self-contained)

Admission runs on **heavypad**, as the `stream-admission` Prefect deployment
defined in the sibling repo
[`zzstoatzz.io/my-prefect-server`](https://tangled.org/zzstoatzz.io/my-prefect-server)
(`flows/stream_admission.py`, `prefect.heavypad.yaml`). Prefect's `heavypad-build-pool` is a
*process* pool, so a flow run there is a bare-metal run on a 32-core box — no
VM tax, no per-run guest setup.

**That deployment is the only sanctioned entry point**, because it is the only
one carrying `concurrency_limit: 1`. On 2026-08-09 two gates ran on the same
box at once (a spindle microVM guest and a bare-metal run) and crossed the
worker cgroup's `memory.high` 534,646 times, silently throttling both. Running
`./scripts/admit run` by hand still works and is fine for debugging a single
suite, but it is invisible to that limit — do not use it as the deploy gate.

`.tangled/workflows/admission.yml` therefore only *triggers* the deployment
(`POST /deployments/<id>/create_flow_run` — the endpoint that sets
`Scheduled`; `POST /flow_runs/` would create an inert `Pending` run). It needs
the `PREFECT_API_URL` and `PREFECT_API_AUTH_STRING` repo secrets.

## the pipeline, in order

| stage | how | notes |
|---|---|---|
| 1. land the change | commit + push to main | `ci.yml` runs fmt + debug/releasesafe tests; admission below is the deploy gate |
| 2. admission | push triggers the `stream-admission` deployment, or trigger it by hand with `sha` + `build_image=true` | bare metal on heavypad; serialised by concurrency_limit=1 |
| 3. publish (optional) | `./scripts/admit publish [sha]` | pins the registry digest into the receipt. Skippable: the registry has been over quota, and deployment verifies by image id instead |
| 4. cut over | flip `STREAM_IMAGE` in `/opt/stream-experiment/.env`, `docker compose up -d stream` | seconds to swap |
| 5. restart recovery | (automatic) | ~4–5 min to serving + dialing upstream; see below |

Notes on the stages:

- **Admission is the gate, not CI.** A receipt (`receipts/<sha>.json`) binds
  one full-suite run to one image digest; `scripts/admit verify <digest>`
  refuses anything without a fully-passing receipt. Never point `.env` at an
  image that `admit verify` rejects.
- **The registry is not on the critical path.** `22bf62c` shipped while atcr
  was over quota by verifying the receipt's recorded `local_id` against the
  image id on the box. `docker save | docker load` preserves image ids, so a
  gate that built on heavypad can still be deployed by transferring it.
- **Uncommitted trees are refused** (`require_clean_tree`) — the image tag is
  the commit, by construction.
- **Partial runs cannot deploy.** `only`/`skip` record every unrun suite as
  `skipped`, and `verify` treats `skipped` exactly like a failure;
  `build_image=false` writes no receipt at all.

## what to expect at restart

- The process replays its cursor from the durable watermark; with
  delete-compaction enabled, startup first rebuilds the live tombstone set.
  Since `fba93d0` (header-level watermark skip, upstream spec §3.4) that
  rebuild reads only segment headers below the watermark and costs seconds to
  minutes, not hours. Before that fix a restart paid a ~3h full-archive scan
  with the tail deaf throughout — if a restart ever takes that long again,
  the skip has regressed; check the rebuild progress lines in the logs.
- **Subscribers are dropped at the restart.** zat-based consumers rotate to
  their next configured host (official jetstreams) and come back on their own
  schedule — they only return to stream when their current host drops them.
  Do not conclude the deploy broke a consumer because it is camping on a
  fallback host.
- The live tail replays the full gap on reconnect; the archive serves
  throughout (the ~30s of docker swap excepted).
- **No consumer bounce is needed any more.** The cold→hot seam wedge that
  made cursor-subscribers deliver in bursts ("jerky, 0↔40/s") was fixed in
  `fca6cc7`; the `22bf62c` cutover confirmed it — coral ran 12 clean samples
  at 20–42/s with `jetstream_subscribe_cold_reads_total` frozen, and no
  consumer was restarted. If burstiness returns, that counter climbing after
  catch-up is the tell, and it is a regression, not an expected cost.

## post-deploy verification (do all four)

1. `curl -s https://stream.waow.tech/metrics | grep stream_build_info` —
   the sha must be the one you shipped.
2. `upstream seq` advancing across two reads ~10s apart (an honest live-tail
   check; `state live` alone only means the process is serving).
3. Delivery cadence: `tools/loadgen` one connection with `--report-s=1` for
   ~60s — per-second deltas, no silent seconds. Run it once live-attach and
   once with `--rewind-s=10` (cursor resume — the path that broke 2026-08-08).
4. Grafana (`grafana.stream.waow.tech`): rate, RSS, and dropped-events panels
   at their usual shapes.

Then commit the ops note: what shipped, why, what was verified. The operator
journal is `docs/ops-changelog.md` in the relay repo.

## when a stage goes wrong

| symptom | do |
|---|---|
| admission suite fails | fix it; there is no skip lever — a `skipped` verdict blocks admission like a failure |
| `admit verify` refuses at cutover | you are deploying an image with no passing receipt; go back to stage 2 |
| restart takes hours, tail deaf | the tombstone-rebuild header skip regressed (see above); the archive still serves — decide rollback vs. wait using the rebuild progress logs |
| consumers stay on fallback hosts | expected; bounce a consumer if you need it back on stream to verify a fix |
| box unreachable mid-deploy | the old container keeps running unless compose already swapped; verify with `docker ps` before assuming an outage |

## operator reference (moved from the root HANDOFF, 2026-08-18)

**Access.** Prod box: `ssh root@89.167.122.160`, compose at
`/opt/stream-experiment/`. heavypad: `ssh stoat@heavypad` over Tailscale (a
hang means re-auth locally, not a dead host); non-login shells need
`PATH=/home/stoat/.local/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/bin`
and `DOCKER_HOST=unix:///var/run/docker.sock`. **heavypad is on the home
LAN: no firehose probes, no bulk transfers through it** — probe from the
prod box instead. Upstream clone lives at
`~/github.com/bluesky-social/jetstream`; keep it at the pin
(re-pin deliberately in campaigns, never chase upstream main).

**Triggering the gate.** Always pass explicit parameters — the stored
deployment defaults are a smoke test, not a contract:

    just -f ~/tangled.org/zzstoatzz.io/my-prefect-server/justfile \
      heavypad-prefect deployment run "stream-admission/stream-admission" \
      --param sha=<sha> --param 'only=[]' --param 'skip=[]' --param build_image=true

The gate self-heals its fixtures: `ensure_simulator` recycles a >12h/wedged
simulator (its getRepo CARs outgrow the oracle window), and
`ensure_upstream` reads the pin from `docs/upstream-harness.md`'s header.
For local iteration on oracle failures,
`STREAM_UPSTREAM_REPO=~/github.com/bluesky-social/jetstream uv run python
tests/differential_oracle.py` gives ~90s cycles against the gate's ~25min.

**Shipping the image.** Preferred: registry push (needs a live
`docker-credential-atcr login` session on the pushing machine). Fallback,
used whenever registry auth is stale — checksummed tar relay, never an
unverified pipe (a dropped `docker save` stream loads "successfully" with
missing layers via content-store dedupe):

    ssh stoat@heavypad '... docker save <tag> > image.tar && sha256sum image.tar'
    scp heavypad:image.tar . && verify sha && scp to box && verify sha && docker load

heavypad has no ssh key to the box, so the relay goes through the
workstation. Deploy = flip `STREAM_IMAGE` in `/opt/stream-experiment/.env`,
`docker compose up -d stream`. Rollback = flip it back.

**Restart semantics.** All maintenance timers are boot-relative: every
restart resets the compaction interval AND the 24h failed-repo retry
timer. Schedule deploys around pending watch points, or accept moving
them. Startup holds data paths at 503 for ~3–4 minutes (storage gate);
the site, /status, and metrics stay up, and one relay-eval window
overlapping the gate will read near-0% — cosmetic and self-explaining.

**Archive keys.** Fleet key: `STREAM_ARCHIVE_API_KEY` in the box `.env`
(never in git, unmetered). Consumer keys: `name:key[:mbps]` lines in
`/opt/stream-experiment/archive-keys.conf` (0600, mounted ro at `/keys/`)
— mint by appending a line, retune by editing the mbps, revoke by
deleting the line; the ring reloads on file mtime within a second, no
restart. See `docs/configuration.md` and `docs/semantic-parity.md`.

**Admission limits.** Three layers; each limit lives at the layer that
can see what it bounds. Source of truth for current values, per layer:

| layer | bounds | read current values | change |
| --- | --- | --- | --- |
| caddy zones (`deploy/site/Caddyfile`) | request rates per client IP/subnet | `grep -B2 -A6 'zone ' deploy/site/Caddyfile` | edit repo copy, `just caddy` (reload; severs nothing) |
| key ring (`/opt/stream-experiment/archive-keys.conf`) | archive bytes/s per key | `curl -s https://stream.waow.tech/metrics \| grep rate_bps` | edit the box file (see Archive keys below) |
| stream process | subscriber count, concurrent cold replays | `grep JETSTREAM_MAX /opt/stream-experiment/.env` (absent/0 = unbounded) | set `JETSTREAM_MAX_SUBSCRIBERS` / `JETSTREAM_MAX_COLD_READERS` in the box `.env`, `docker compose up -d stream` |

Values as first deployed (2026-08-21, untested guesses — retune from
metrics, not from this table):

| control | value | basis |
| --- | --- | --- |
| ws connect attempts | 30 / 5m per /24 | reconnect loops are rare in healthy clients |
| keyed xrpc requests | 15,000 / min per IP | a 32 MB/s consumer ≈ 150 getBlock/s (~214 KB avg block); floor is highest observed legit rate + headroom |
| anonymous http | 240 / min per /24 | static site + probes; also absorbs 401 spray |
| max subscribers | unset (unbounded) | pending first load test |
| max cold readers | unset (unbounded) | pending first load test |

Retuning inputs: `stream_archive_key_*` series (per-key request/byte
rates), `stream_subscribe_capacity_rejects_total`,
`stream_subscribe_cold_contended_total`, caddy logs (429 counts per
zone). When a limit fires against legitimate traffic, the fix is a new
row here with a new date and basis, not a silent edit.

The caddy image is custom (`stream-caddy:rl`, `deploy/caddy/Dockerfile`:
caddy 2.10 + `mholt/caddy-ratelimit`) — stock caddy has no rate
limiter. Image rebuild: `docker build -t stream-caddy:rl -f
caddy.Dockerfile .` on the box, then `docker compose up -d caddy`; a
recreate severs live subscriber websockets once (clients reconnect).
Caddyfile-only changes go through `just caddy` (reload; severs
nothing). Cap semantics and upstream contrast:
`docs/semantic-parity.md`.

**Namespace doctrine (settled).** We serve upstream's
`network.bsky.jetstream.*` NSIDs on purpose — the NSID names Bluesky's
schema contract, not the server. Stream-only extensions, if ever added,
get `dev.zat.stream.*`. Rationale: `docs/product-copy-context.md`.

## Strata playground

`just strata` publishes https://stream.waow.tech/strata/
independently of the homepage, Worker, and Stream binary. It builds the
`strata/` source (run `bun install` there once for its pinned esbuild) and
fetches one public summary snapshot during the deploy. Caddy serves only static files, including gzip sidecars. No backend,
credential, archive sampling endpoint, or scheduled refresh is deployed.
Historical sampling stays local; the public playground offers the map, search,
profile avatars, and canvas pan/zoom. The former live-event feed has been
removed; the playground opens no WebSocket connections.

The storage comparison uses `strata/storage-report.json`, embedded in the
static summary at deploy time. Refresh it manually with
`uv run strata/inspect-storage.py` before deploying when needed; its scope and
limits are in [strata-storage-discovery.md](strata-storage-discovery.md).
That bounded inspection reads file listings, headers, and indexes using the
existing archive credential locally; it does not download record bodies.
Deploying does not run the inspection. Visitors trigger no archive reads or
Microcosm API calls; UFOs integration is a contextual external link.

The homepage footer links to `/strata/`. The former `/_strata-preview/` route
redirects there, including asset paths. Existing `/strata/api` endpoints still
proxy to the summary Worker for compatibility. The route definition is in
`deploy/site/Caddyfile`; normal updates need no Caddy reload. `just strata-preview`
remains an alias for `just strata`.
Uploads remain in `site/.strata-preview-releases/<UTC timestamp>` on the existing
host, selected by the atomically replaced `site/strata` symlink. Rollback selects
an earlier release with the same temporary-symlink/rename operation in
`scripts/deploy-strata-preview`. The historical release-directory and script
names are retained; the public URL and deployment output are `/strata/`.

Profile recognition is also embedded in the static summary.
`python3 strata/refresh-identities.py` refreshes the shared catalog manually, using batches of 25 handles with a 100-request ceiling, a pause between
batches, and no retry loop. Successful matches expire after seven days; missing
profiles after one day. It stops on an HTTP error, including rate limiting, and
preserves completed batches. Deploy does not refresh this catalog. Visitors
make no profile API calls; avatar images use their source's HTTP cache and a
bounded canvas drawing cache. No avatar blobs are stored on the Stream disk.

### Slow-connection loading

The public HTML embeds only source totals and recognition metadata. The full
record-type catalogue is grouped by namespace and loaded once on demand when a
visitor selects a source or searches. Counts and record types remain exact.
The build bundles the scripts with the sibling repo's installed esbuild, and
content-hashes script, stylesheet, and catalogue filenames. Caddy caches these
immutable assets for one year; prior asset versions are retained during release
switches so a cached page continues to work. HTML is fresh for 60 seconds, with
stale-while-revalidate and stale-if-error allowances. This is HTTP caching, not
a guarantee of offline availability. Detail requests time out after 15 seconds
and can be retried by selecting a source or searching again.

For a local build without deploying, run `python3 scripts/deploy-strata-preview
--output-dir /tmp/strata-build`. `--overview PATH` reuses a saved overview instead
of fetching the public summaries. Avatar resolution and archive inspection are
still manual; deployment adds no service or scheduled job.

## eMudhra remediation (2026-10-21)

The runtime and builder require Debian `ca-certificates` at least
`20250419~deb12u2`. The shared zat/websocket dependencies carry explicit
compromised-key distrust, and the direct backfill `std.http.Client` applies
that same policy before requests, including HTTP-to-HTTPS redirects.
The OS bundle also covers libcurl telemetry. Restart the container after
rebuilding so all clients load the updated trust.

`zig build tls-probe` exercises the actual backfill fetch path against the
three published negative hosts and a valid HTTPS control. The image includes
`/usr/local/bin/stream-tls-probe` for the same check after admission. Zig's
HTTP API collapses handshake failures to `TlsInitializationFailed`; corroborate
these refusals with strict OpenSSL certificate probes in the runtime and the
shared transport fixture tests. A failed control makes the probe fail.

This change still requires the complete serialized Prefect admission run and
its receipt before cutover. Production before this change was `d2b13ec`, image
ID `sha256:81c49e990b88dfb2c59048fd44ac4d4f4f72bf7fe44489d341217d3c792625ea`.
The intervening main commit `d1a3118` changes only the separately deployed
Strata Worker. Preserve the old image for availability rollback; it retains
vulnerable trust.

The Caddy rebuild preserves Caddy 2.10.2 and caddy-ratelimit v0.1.0 while
using Go 1.26.10 and Alpine CA certificates at least `20261020-r0`. Build
`deploy/caddy/Dockerfile` under a new tag, validate the existing Caddyfile and
check `caddy build-info` before changing the compose image. Recreating Caddy
severs existing WebSockets once. The pre-remediation `stream-caddy:rl` image
ID is `sha256:2f553afd95c320ce5c603549b1a7e6943db260ac09129e14266d72aa078045db`;
preserve that ID before any tag reuse.