very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall

[crawler] keep reading pages while one waits for the firehose

the crawl awaited each page whose newer revs waited for the firehose, so on a caught-up box any page with an account that committed in the last few seconds cost the margin and the lag before the next page was read, hours over a full pass, and during a long catch-up one page could hold the crawl for the whole timeout. a page's newer listings and its cursor now go into a bounded line of waiting pages (32), which a task of its own hands to the worker in the order the pages were read, each once the firehose caught up to it or its timeout passed. the rest of each page goes to the worker right away, the crawl only waits when the line is full, and a page's cursor still never moves before every page ahead of it is in.


Author ptr.pet
Date Commit Parent 1c6ef977
+271 -87
3 changed files
+271 -87
3 changed files