diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 427603b..e37d70a 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -115,6 +115,33 @@ extended. Previous envelope: 24 h, $50, ~$2.53–$18 actual. - Recompute the operator `/32`; never reuse the one in `.experiment.env`. - `./scripts/admit verify ` must admit the artifact you intend to run. - Confirm the artifact matches the commit you think it does. +- **Feasibility: can this run finish at all?** Measure, do not assume. + +### Stage 0b — the feasibility check (~15 min, €0.03) + +Every experiment through 5 was scoped to finish the network in 24 hours, and +none of them could have. The gate has twenty correctness suites and none of +them asks how long the job takes, so an artifact can be 20/20 admitted, every +row verified, and still be forty days from finishing a one-day experiment. + +Before provisioning, on a throwaway box against the real network: + +1. measure sustained **repos/s** and **bytes/repo** (10 minutes of crawling); +2. get the network size — binary-search the `listRepos` cursor until a page + comes back empty; +3. compute `size x bytes-per-repo / link-rate` and compare it with the + deadline. + +Measured 2026-07-28: **~22.7M repos, ~9.5 MB each, ~216 TB**. At the 484 Mbps +a CCX43 sustains that is **41 days**; at a perfect 1 Gbps it is **20 days**. +Upstream's documented ~16 hours implies ~30 Gbps and ~394 repos/s, which is +infrastructure, not implementation — no code moves 216 TB through one NIC in a +day. + +**If the projection exceeds the deadline, do not provision.** Either scale the +goal to what the link can carry (a bounded, honest slice) or change the +infrastructure. Running anyway produces a partial archive and calls it a +failure of the code. **If it fails:** stop. Do not provision to investigate. diff --git a/docs/semantic-parity.md b/docs/semantic-parity.md index 2fb1b62..878e0ec 100644 --- a/docs/semantic-parity.md +++ b/docs/semantic-parity.md @@ -42,8 +42,17 @@ test, and this audit must be reviewed together. ## Current decision -**No row is blocked. One row was marked verified and was wrong, and the -correction is the most important thing on this page.** +**One row is blocked, and it is not a code defect: a whole-network backfill +cannot finish on the hardware every experiment has used.** + +~22.7M repositories at ~9.5 MB each is ~216 TB, which is ~41 days at the +484 Mbps a CCX43 sustains and ~20 days at a perfect 1 Gbps. Every experiment +so far was scoped to 24 hours. This audit carried twenty correctness suites and +never asked how long the work takes, so artifacts were admitted as ready while +being weeks from finishing. Correctness evidence is not feasibility evidence. + +**A second row was marked verified and was wrong, and that correction is the +other thing worth reading here.** Experiment 4 (27 July) ran 55 minutes and durably committed nothing: 12,686 repositories fetched and emitted, zero ever queued for completion. The cause @@ -82,6 +91,7 @@ The detailed bootstrap audit is in | Surface | Status | What is actually established | Blocking or missing evidence | |---|---|---|---| +| Whole-network feasibility | **blocked** | Measured 2026-07-28 against bsky.network: ~22.7M repositories (binary search on the `listRepos` cursor), ~9.5 MB mean transfer per repository, ~216 TB for a full pass. A CCX43 sustains ~484 Mbps across 100 workers, giving ~6 repos/s. | **A full-network backfill cannot complete on a single Hetzner box.** 216 TB is ~41 days at the measured rate and ~20 days at a perfect 1 Gbps. Upstream's documented ~16 hours implies ~30 Gbps sustained and ~394 repos/s, which is infrastructure rather than implementation. Nothing in this audit measured throughput before experiment 5, so five experiments were scoped to a target physics rules out. Either the goal shrinks to a bounded slice the link can carry, or the infrastructure changes; no code change closes this row. | | Lifecycle phase machine | **verified** | Phase reads distinguish absence from RocksDB failure, reject unknown values, and use synced writes. Bootstrap writes `merging` after backfill drains while bootstrap-live is still running, exactly at the pinned upstream boundary; process-kill crashpoints exercise both sides of that write. Cleanup writes `steady_state` only after directory sync and cursor deletion. | Exact persisted phase-entry timestamps are a separate status-surface gap; this row establishes transition state and ordering only. | | Whole-network bootstrap | **partial** | listRepos pages form a page-aligned dispatch/checkpoint unit; eligible repositories are shuffled; download concurrency and CAR preparation are real. Successful repositories become durable independently at archive-writer durability boundaries. Cleanup at every preparation/emission ownership transfer is the parse arena's by construction, and an allocation-failure sweep asserts exhaustion always surfaces as `OutOfMemory`. The batch submit/consume deadlock below is fixed and the fix is confirmed against the real network. | This row was marked **verified** and was wrong. `finishWorkBatch` submitted an entire 100,000-job batch before consuming a result, against a queue of `worker_count * 2`; with workers blocked in `Budget.acquire` (which waits rather than failing) and the budget released only by the main fiber consuming results, bootstrap deadlocked with zero repositories ever queued for completion. Experiment 4 ran 55 minutes and committed nothing. Fixed in `fcc63f8`, reproduced and verified on a ccx23 against bsky.network -- 1,403 handled / 0 durable before, `queued == handled` after. It stays **partial** because no offline suite reaches it: the pinned simulator's repositories are ~4 KiB, so concurrent demand cannot exceed a budget one repository fits in. Only a real-network run confirms this class. | | Merge and post-bootstrap discovery | **verified** | Source rows are filtered by the backfill revision; source cursor and latest-revision updates share one synced batch (`commitMergeSource`, which refreshes `latest_rev`/`updated_us` while preserving the backfill `rev` — asserted by the real-merge test); merge cursor corruption/read errors fail closed; the restart guard treats only `FileNotFound` as cleanup-complete; discovery records active and inactive unknown repositories and follows arbitrary-length cursors with loop detection; successful cleanup is directory-synced. Every source-file path fails closed — `readFileAlloc`, checksum-verifying `Sealed.parse`, `readBlock` and the destination append all propagate, with no `catch` swallow — and a missing source segment raises `SourceIndexGap` rather than being skipped; a physical test deletes the middle of three sealed sources and asserts the error, where without the guard merge consumes 2 of 3 sources and reports success. The pending pass is now upstream's `RunPendingRepoRetryPass` exactly: the same bounded runner with `eligible_status` flipped to pending, so it inherits the worker pool, per-host gate, computed backoff and 429 host parking rather than reimplementing a weaker version. A test asserts a failed pending repository gets upstream's `RecordRetryFailure` bookkeeping — status failed, `attempts` and `retry_count` incremented, `next_attempt_us` in the future — and that a failed sibling is untouched, which the previous hand-rolled loop got wrong on the last two fields. | No known divergence remains. The runner is shared with the steady retry loop, so that row's persisted-host-parking caveat applies here too; it is tracked there rather than duplicated. |