From 14bfd4e1ba2de8135e742e109677e22005b25794 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Mon, 27 Jul 2026 21:11:37 -0500 Subject: [PATCH] gate: a whole-network backfill cannot finish on the hardware we use Measured against bsky.network: ~22.7M repositories (binary search on the listRepos cursor), ~9.5 MB mean transfer each, ~216 TB for a full pass. A CCX43 sustains ~484 Mbps across 100 workers -- about 41 days. A perfect 1 Gbps is still 20. Upstream's documented ~16 hours implies ~30 Gbps and ~394 repos/s, which is infrastructure, not implementation. Every experiment through 5 was scoped to 24 hours and none could have finished. This audit carried twenty correctness suites and never asked how long the work takes, so artifacts were admitted as ready while being weeks from completion. Correctness evidence is not feasibility evidence. Adds the row as blocked, and a Stage 0b feasibility check to the runbook: measure repos/s and bytes/repo on a throwaway box, get the network size, and compare the projection with the deadline before provisioning anything. Co-Authored-By: Claude Opus 5 (1M context) --- docs/deployment-runbook.md | 27 +++++++++++++++++++++++++++ docs/semantic-parity.md | 14 ++++++++++++-- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 427603b..e37d70a 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -115,6 +115,33 @@ extended. Previous envelope: 24 h, $50, ~$2.53–$18 actual. - Recompute the operator `/32`; never reuse the one in `.experiment.env`. - `./scripts/admit verify ` must admit the artifact you intend to run. - Confirm the artifact matches the commit you think it does. +- **Feasibility: can this run finish at all?** Measure, do not assume. + +### Stage 0b — the feasibility check (~15 min, €0.03) + +Every experiment through 5 was scoped to finish the network in 24 hours, and +none of them could have. The gate has twenty correctness suites and none of +them asks how long the job takes, so an artifact can be 20/20 admitted, every +row verified, and still be forty days from finishing a one-day experiment. + +Before provisioning, on a throwaway box against the real network: + +1. measure sustained **repos/s** and **bytes/repo** (10 minutes of crawling); +2. get the network size — binary-search the `listRepos` cursor until a page + comes back empty; +3. compute `size x bytes-per-repo / link-rate` and compare it with the + deadline. + +Measured 2026-07-28: **~22.7M repos, ~9.5 MB each, ~216 TB**. At the 484 Mbps +a CCX43 sustains that is **41 days**; at a perfect 1 Gbps it is **20 days**. +Upstream's documented ~16 hours implies ~30 Gbps and ~394 repos/s, which is +infrastructure, not implementation — no code moves 216 TB through one NIC in a +day. + +**If the projection exceeds the deadline, do not provision.** Either scale the +goal to what the link can carry (a bounded, honest slice) or change the +infrastructure. Running anyway produces a partial archive and calls it a +failure of the code. **If it fails:** stop. Do not provision to investigate. diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index 2fb1b62..878e0ec 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -42,8 +42,17 @@ test, and this audit must be reviewed together. ## Current decision -**No row is blocked. One row was marked verified and was wrong, and the -correction is the most important thing on this page.** +**One row is blocked, and it is not a code defect: a whole-network backfill +cannot finish on the hardware every experiment has used.** + +~22.7M repositories at ~9.5 MB each is ~216 TB, which is ~41 days at the +484 Mbps a CCX43 sustains and ~20 days at a perfect 1 Gbps. Every experiment +so far was scoped to 24 hours. This audit carried twenty correctness suites and +never asked how long the work takes, so artifacts were admitted as ready while +being weeks from finishing. Correctness evidence is not feasibility evidence. + +**A second row was marked verified and was wrong, and that correction is the +other thing worth reading here.** Experiment 4 (27 July) ran 55 minutes and durably committed nothing: 12,686 repositories fetched and emitted, zero ever queued for completion. The cause @@ -82,6 +91,7 @@ The detailed bootstrap audit is in | Surface | Status | What is actually established | Blocking or missing evidence | |---|---|---|---| +| Whole-network feasibility | **blocked** | Measured 2026-07-28 against bsky.network: ~22.7M repositories (binary search on the `listRepos` cursor), ~9.5 MB mean transfer per repository, ~216 TB for a full pass. A CCX43 sustains ~484 Mbps across 100 workers, giving ~6 repos/s. | **A full-network backfill cannot complete on a single Hetzner box.** 216 TB is ~41 days at the measured rate and ~20 days at a perfect 1 Gbps. Upstream's documented ~16 hours implies ~30 Gbps sustained and ~394 repos/s, which is infrastructure rather than implementation. Nothing in this audit measured throughput before experiment 5, so five experiments were scoped to a target physics rules out. Either the goal shrinks to a bounded slice the link can carry, or the infrastructure changes; no code change closes this row. | | Lifecycle phase machine | **verified** | Phase reads distinguish absence from RocksDB failure, reject unknown values, and use synced writes. Bootstrap writes `merging` after backfill drains while bootstrap-live is still running, exactly at the pinned upstream boundary; process-kill crashpoints exercise both sides of that write. Cleanup writes `steady_state` only after directory sync and cursor deletion. | Exact persisted phase-entry timestamps are a separate status-surface gap; this row establishes transition state and ordering only. | | Whole-network bootstrap | **partial** | listRepos pages form a page-aligned dispatch/checkpoint unit; eligible repositories are shuffled; download concurrency and CAR preparation are real. Successful repositories become durable independently at archive-writer durability boundaries. Cleanup at every preparation/emission ownership transfer is the parse arena's by construction, and an allocation-failure sweep asserts exhaustion always surfaces as `OutOfMemory`. The batch submit/consume deadlock below is fixed and the fix is confirmed against the real network. | This row was marked **verified** and was wrong. `finishWorkBatch` submitted an entire 100,000-job batch before consuming a result, against a queue of `worker_count * 2`; with workers blocked in `Budget.acquire` (which waits rather than failing) and the budget released only by the main fiber consuming results, bootstrap deadlocked with zero repositories ever queued for completion. Experiment 4 ran 55 minutes and committed nothing. Fixed in `fcc63f8`, reproduced and verified on a ccx23 against bsky.network -- 1,403 handled / 0 durable before, `queued == handled` after. It stays **partial** because no offline suite reaches it: the pinned simulator's repositories are ~4 KiB, so concurrent demand cannot exceed a budget one repository fits in. Only a real-network run confirms this class. | | Merge and post-bootstrap discovery | **verified** | Source rows are filtered by the backfill revision; source cursor and latest-revision updates share one synced batch (`commitMergeSource`, which refreshes `latest_rev`/`updated_us` while preserving the backfill `rev` — asserted by the real-merge test); merge cursor corruption/read errors fail closed; the restart guard treats only `FileNotFound` as cleanup-complete; discovery records active and inactive unknown repositories and follows arbitrary-length cursors with loop detection; successful cleanup is directory-synced. Every source-file path fails closed — `readFileAlloc`, checksum-verifying `Sealed.parse`, `readBlock` and the destination append all propagate, with no `catch` swallow — and a missing source segment raises `SourceIndexGap` rather than being skipped; a physical test deletes the middle of three sealed sources and asserts the error, where without the guard merge consumes 2 of 3 sources and reports success. The pending pass is now upstream's `RunPendingRepoRetryPass` exactly: the same bounded runner with `eligible_status` flipped to pending, so it inherits the worker pool, per-host gate, computed backoff and 429 host parking rather than reimplementing a weaker version. A test asserts a failed pending repository gets upstream's `RecordRetryFailure` bookkeeping — status failed, `attempts` and `retry_count` incremented, `next_attempt_us` in the future — and that a failed sibling is untouched, which the previous hand-rolled loop got wrong on the last two fields. | No known divergence remains. The runner is shared with the steady retry loop, so that row's persisted-host-parking caveat applies here too; it is tracked there rather than duplicated. | -- knotmirror 0.0.1