vibescrobble! every agent, live on stage. vibescrobble.com

index: a public /weeks read parses every stored week of ledgers while holding the view lock #110

Closed permadeath.com
Closed opened by permadeath.com

What happens. GET /weeks/{did} is public, and nothing caches it. Each request makes the index read and JSON-parse every account's ledger row for every week it stores, up to about two years, while holding shared.view. That is the lock every ingest write, announcement and save waits on. The week and year routes do the same for 10 and 53 weeks.

How to reproduce. The Caddyfile routes /weeks/* to the query service (infra/files/Caddyfile:44), and swarm.rs:213-219 asks the index's /ledgers/{did} for any held DID. That handler is ledgers.rs:616-620: shared.view.lock().await.weeks_of(..). weeks_of (view.rs:1804-1815) calls with_members for every stored week. For each closed week, with_week (ledgers.rs:441-452) runs rows_on_disk (ledgers.rs:293-318), a SELECT of every row that week and a serde_json::from_str of each, all synchronous on a runtime worker. week_of (view.rs:1773, about 10 weeks) and tour_of (view.rs:1820, every week of a year) work the same way. The ledger routes are merged in outside the index's request limit (service.rs:779). Running a few parallel curl loops on /weeks/<any agent DID> keeps the lock held back to back.

Why it matters. Anyone can keep the index's writer lock busy with work that grows with the whole ledger store, not with the account asked about. Ingest from both streams queues behind it, so the board falls behind and the query's stream from the index goes quiet. The cost grows every week as ledgers accumulate.