dev.permadeath.com
unified development activity
Star
Fork
Watch
dev.permadeath.com docs GLOBAL-RERUN.md
4.4 kB · 120 lines
Markdown
at main

Global crawl rebuild and re-run #

Prerequisite: merge PRs #17 and #18 before rebuilding. They finish the imported-history, fork, review-candidate, excludes-file and filter-only work.

What changed #

The global task is unchanged in shape: one crawl --global process with its own SQLite state and summary. It still never publishes the site.

New behavior:

  • skips fork records
  • drops commits older than TALLY_GLOBAL_IMPORTED_BEFORE and older than each repo's Tangled creation date minus TALLY_GLOBAL_REPO_AGE_GRACE_DAYS
  • flags repos whose default branch exceeds TALLY_GLOBAL_REVIEW_REPO_COMMITS and keeps walking them
  • stops at TALLY_GLOBAL_MAX_PAGES as a per-repo backstop
  • logs progress every 10 minutes or 250 repos
  • uploads a checkpoint after every 30 minutes of crawling
  • writes a partial summary at each checkpoint and a complete one at the end (a shard writes its counted changes instead; global-merge makes the summary)
  • gives up on one repository after TALLY_REPO_TIMEOUT_SECS
  • applies TALLY_GLOBAL_EXCLUDES (global-excludes.txt by default)
  • removes excluded/imported refs from the summary even in an older database

Before the full run #

If a state/global.sqlite3 already exists, audit it first:

aws s3 cp s3://STATE_BUCKET/state/global.sqlite3 /tmp/global.sqlite3

TALLY_DB=/tmp/global.sqlite3 \
TALLY_OUT=/tmp/tally-audit-out \
TALLY_GLOBAL_SUMMARY_OUT=/tmp/global-summary.json \
TALLY_PUBLISH=false \
TALLY_GLOBAL_REVIEW_REPO_COMMITS=20000 \
TALLY_GLOBAL_EXCLUDES=global-excludes.txt \
TALLY_GLOBAL_REPO_AGE_GRACE_DAYS=30 \
cargo run --release -- crawl --global --filter-only

Expected: repos_queued=0, commits_seen=0, an excluded-fork count, review candidates for indexed repos over 20k, and a partial:true summary. The audit only knows about commits already indexed; a giant repo that was never reached is caught by the full run's total probe.

A fresh global state is cleaner for the first filtered run. If reusing the old one, keep the audit summary only as a check; the full run rewrites it.

Full run environment #

Use the global task's own database and state key. Do not share them with the hourly task.

TALLY_DB=/data/state.sqlite3
TALLY_STATE_KEY=state/global.sqlite3
TALLY_STATE_BUCKET=STATE_BUCKET
TALLY_SITE_BUCKET=SITE_BUCKET
TALLY_PUBLISH=true
TALLY_GLOBAL_SUMMARY_OUT=s3://STATE_BUCKET/state/global-summary.json

TALLY_GLOBAL_SKIP_FORKS=true
TALLY_GLOBAL_SKIP_DESCRIBED_COPIES=true
TALLY_BRANCH_SCOPE=default
TALLY_GLOBAL_IMPORTED_BEFORE=2025-01-01
TALLY_GLOBAL_REPO_AGE_GRACE_DAYS=30
TALLY_GLOBAL_MAX_PAGES=40
TALLY_GLOBAL_REVIEW_REPO_COMMITS=20000
TALLY_GLOBAL_EXCLUDES=global-excludes.txt

TALLY_GLOBAL_CONCURRENCY=4
TALLY_GLOBAL_PER_HOST_RPS=2
TALLY_GLOBAL_HTTP_ATTEMPTS=3
TALLY_GLOBAL_PROGRESS_SECS=600
TALLY_GLOBAL_PROGRESS_REPOS=250
TALLY_GLOBAL_CHECKPOINT_SECS=1800
TALLY_GLOBAL_CHECKPOINT_REPOS=1000
TALLY_GLOBAL_MIN_COVERAGE_PCT=90
TALLY_CIRCUIT_FAILURES=5
TALLY_CIRCUIT_COOLDOWN_SECS=60

TALLY_GLOBAL_REFRESH_SECS=2592000
TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS=35

TALLY_PUBLISH=true makes the global task upload its own SQLite snapshot and summary; it does not call the site publisher.

Hourly task #

Keep the existing owned task and point it at the new summary:

TALLY_GLOBAL_SUMMARY=s3://STATE_BUCKET/state/global-summary.json

It publishes the site. If the global summary is missing, stale, or partial below 90% coverage, the share is hidden; a usable partial summary is labelled with its coverage.

What to watch #

  • Progress lines: repos_done, repos_queued, repos_total, commits_seen, commits_new, failures, elapsed_secs.
  • Checkpoint lines: wrote global summary and published index state.
  • global review candidate when a repo exceeds 20k; investigate and add its DID to global-excludes.txt if it is an import, then re-audit.
  • global crawl stopped at the page cap when the page backstop fires.
  • A final summary with partial:false and repos_reached / repos_total at or above the configured coverage.
  • If the share is hidden, inspect the summary's coverage before assuming the crawl failed.

Done #

A good run has a complete summary, coverage high enough to show the share, no large imported/test repo in the top-project list, and the hourly dashboard reading the new summary without a site redeploy.