panoptiblog no longer walks every repository every day. A full walk seeds a gzipped record set in S3; daily runs read Jetstream commits for the four standard.site collections, re-read only the repositories those commits name, and rebuild the dataset from the set. A full walk reconciles the whole set every two weeks, and a run has a four-hour deadline. The crawl panel distinguishes incremental runs from full ones.
The record set is saved before the dataset, so a failed upload can rebuild without another network walk. Missing, unreadable, stale, method-mismatched, or retention-shifted state falls back to a full walk; so does a silent Jetstream. Canaries and failed repositories are retried every incremental run. Transient repository 404s and handle-lookup failures keep the previous records, deleted accounts are removed immediately, and only event-driven failures can gate a run. The task role gains GetObject and the task moves to 4 GB.
Verification: npx vitest run scripts/panoptiblog passes 150 tests and npm run check passes. Live full and incremental runs over 200 repositories produced identical publication rows, scores, and ranks; two smaller full/incremental pairs, including one after the method-version change and one on the final code, also matched exactly. Deploying the image runs one full walk to seed the state; Terraform must be applied for the IAM and memory changes.
🤖 Generated with Claude Code