# Global crawl rebuild and re-run

Prerequisite: merge PRs #17 and #18 before rebuilding. They finish the imported-history,
fork, review-candidate, excludes-file and filter-only work.

## What changed

The global task is unchanged in shape: one `crawl --global` process with its own
SQLite state and summary. It still never publishes the site.

New behavior:

- skips fork records
- drops commits older than `TALLY_GLOBAL_IMPORTED_BEFORE` and older than each
  repo's Tangled creation date minus `TALLY_GLOBAL_REPO_AGE_GRACE_DAYS`
- flags repos whose default branch exceeds
  `TALLY_GLOBAL_REVIEW_REPO_COMMITS` and keeps walking them
- stops at `TALLY_GLOBAL_MAX_PAGES` as a per-repo backstop
- logs progress every 10 minutes or 250 repos
- uploads a checkpoint after every 30 minutes of crawling
- writes a partial summary at each checkpoint and a complete one at the end
  (a shard writes its counted changes instead; `global-merge` makes the summary)
- gives up on one repository after `TALLY_REPO_TIMEOUT_SECS`
- applies `TALLY_GLOBAL_EXCLUDES` (`global-excludes.txt` by default)
- removes excluded/imported refs from the summary even in an older database

## Before the full run

If a `state/global.sqlite3` already exists, audit it first:

```bash
aws s3 cp s3://STATE_BUCKET/state/global.sqlite3 /tmp/global.sqlite3

TALLY_DB=/tmp/global.sqlite3 \
TALLY_OUT=/tmp/tally-audit-out \
TALLY_GLOBAL_SUMMARY_OUT=/tmp/global-summary.json \
TALLY_PUBLISH=false \
TALLY_GLOBAL_REVIEW_REPO_COMMITS=20000 \
TALLY_GLOBAL_EXCLUDES=global-excludes.txt \
TALLY_GLOBAL_REPO_AGE_GRACE_DAYS=30 \
cargo run --release -- crawl --global --filter-only
```

Expected: `repos_queued=0`, `commits_seen=0`, an excluded-fork count, review
candidates for indexed repos over 20k, and a `partial:true` summary. The audit only knows about commits already indexed; a
giant repo that was never reached is caught by the full run's total probe.

A fresh global state is cleaner for the first filtered run. If reusing the old
one, keep the audit summary only as a check; the full run rewrites it.

## Full run environment

Use the global task's own database and state key. Do not share them with the
hourly task.

```text
TALLY_DB=/data/state.sqlite3
TALLY_STATE_KEY=state/global.sqlite3
TALLY_STATE_BUCKET=STATE_BUCKET
TALLY_SITE_BUCKET=SITE_BUCKET
TALLY_PUBLISH=true
TALLY_GLOBAL_SUMMARY_OUT=s3://STATE_BUCKET/state/global-summary.json

TALLY_GLOBAL_SKIP_FORKS=true
TALLY_GLOBAL_SKIP_DESCRIBED_COPIES=true
TALLY_BRANCH_SCOPE=default
TALLY_GLOBAL_IMPORTED_BEFORE=2025-01-01
TALLY_GLOBAL_REPO_AGE_GRACE_DAYS=30
TALLY_GLOBAL_MAX_PAGES=40
TALLY_GLOBAL_REVIEW_REPO_COMMITS=20000
TALLY_GLOBAL_EXCLUDES=global-excludes.txt

TALLY_GLOBAL_CONCURRENCY=4
TALLY_GLOBAL_PER_HOST_RPS=2
TALLY_GLOBAL_HTTP_ATTEMPTS=3
TALLY_GLOBAL_PROGRESS_SECS=600
TALLY_GLOBAL_PROGRESS_REPOS=250
TALLY_GLOBAL_CHECKPOINT_SECS=1800
TALLY_GLOBAL_CHECKPOINT_REPOS=1000
TALLY_GLOBAL_MIN_COVERAGE_PCT=90
TALLY_CIRCUIT_FAILURES=5
TALLY_CIRCUIT_COOLDOWN_SECS=60

TALLY_GLOBAL_REFRESH_SECS=2592000
TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS=35
```

`TALLY_PUBLISH=true` makes the global task upload its own SQLite snapshot and
summary; it does **not** call the site publisher.

## Hourly task

Keep the existing owned task and point it at the new summary:

```text
TALLY_GLOBAL_SUMMARY=s3://STATE_BUCKET/state/global-summary.json
```

It publishes the site. If the global summary is missing, stale, or partial
below 90% coverage, the share is hidden; a usable partial summary is labelled
with its coverage.

## What to watch

- Progress lines: `repos_done`, `repos_queued`, `repos_total`,
  `commits_seen`, `commits_new`, `failures`, `elapsed_secs`.
- Checkpoint lines: `wrote global summary` and `published index state`.
- `global review candidate` when a repo exceeds 20k; investigate and add its
  DID to `global-excludes.txt` if it is an import, then re-audit.
- `global crawl stopped at the page cap` when the page backstop fires.
- A final summary with `partial:false` and `repos_reached / repos_total` at or
  above the configured coverage.
- If the share is hidden, inspect the summary's coverage before assuming the
  crawl failed.

## Done

A good run has a complete summary, coverage high enough to show the share, no
large imported/test repo in the top-project list, and the hourly dashboard
reading the new summary without a site redeploy.