# dev.permadeath.com **tally** is the deduplicated commit-activity service behind [dev.permadeath.com](https://dev.permadeath.com). Repository: [tangled.org/permadeath.com/dev.permadeath.com](https://tangled.org/permadeath.com/dev.permadeath.com). Tally answers four questions in one place: 1. How much have I committed on Tangled, per identity? 2. How much have my didbot agents committed there? 3. How much have I committed on GitHub, including private and work repositories? 4. What fraction of all Tangled commits is mine? The site is static. A scheduled Rust worker crawls, writes JSON and HTML, and publishes them to S3 behind CloudFront. There is no public write path and no server process in the request path. ## Preview ![Tally dashboard, dark](docs/dashboard-dark.png) The interactive, self-contained snapshot is [docs/mockup.html](docs/mockup.html); open it in a browser to use the period selector, table view and theme toggle. The screenshots contain real activity data and are intentionally committed. Rebuild it from a built site with `python3 scripts/build_mockup.py dist/data/site.json`. The heatmap's day panel fetches `data/days/YYYY-MM.json`, so it needs the built site behind a local HTTP server (`python3 -m http.server -d dist`); the mockup can't load it. Each build also renders the link preview: `og.png` (by default four of the page's charts as wordless red shapes: the 90-day calendar, the 24 h wheel, written-to-landed and project timelines; `TALLY_OG_LAYOUT=year` brings back the year's matrix) and its alt text, with the 30-day count of the owner's and their agents' commits together. The meta tags' title and description are fixed text. The image URL carries a fingerprint of the PNG, so a crawler that caches by URL refetches it only when the card changed. ## Prior Tangled census The earlier network research lives at [blogposts/tangled-stats](https://tangled.org/permadeath.com/blogposts/tree/main/tangled-stats). As of 4 September 2026 its closure crawl reached **182 knots, 14,528 repositories and 17,939 accounts**. It counted commits from merged-PR patches: **278,018 all-time**, of which permadeath.com was **3,014 (1.1%)**, and **16,966 / 2,426 (14.3%)** for August 2026. Its own notes say commits pushed straight to a branch are not counted and that pull-record shapes changed over the crawl. Tally takes the branch-walk route instead: every branch is read, direct pushes are included, and commits collapse by SHA and `Change-Id`. Its denominator is therefore larger and its share is **not directly comparable** to the patch-derived census. The dashboard labels both the definition and the current index coverage. ## Commit definition One definition applies to Tangled, GitHub and the redacted work inbox: > A logical change is a distinct `Change-Id:` when the commit carries one > (atgc writes these trailers), otherwise a distinct Git object ID (SHA). A > commit counts once its repository's default branch reaches it (merged work; > `TALLY_BRANCH_SCOPE=all` counts every branch instead), and counts once > regardless of how many branches, resubmits, rebases, tags, or forges report > it. Exact resubmits have the same SHA and are already excluded. A rebase keeps > the Change-Id, so the rewritten object collapses into the same logical change. > The same SHA on Tangled and GitHub is one change. Merge commits count once. In > the share, a rebase without a Change-Id counts once when its author, author > time and subject match; elsewhere it remains a separate object. > Tags are not counted separately. Attribution uses the Git author, and dates > are author dates in UTC. The dashboard also shows the raw unique Git-object > count for context. Tangled's index is built from ref updates, so the same commit can be announced on every branch and again on every resubmit of a pull request. A smoke crawl of 17 repositories read 130,690 branch-reported commits and found 6,680 distinct commits — about 19.6x inflation. Tally walks every branch, stops when a branch joins history it has already indexed, and stores one row per SHA. The GitHub and work crawlers walk branches the same way and deduplicate by SHA; co-authored commits count for their primary author. - A commit reachable from five branches is one row. - A commit seen on Tangled and mirrored on GitHub is one row. - Author dates are used, not push dates; the committer is often `Tangled ` with the time of the push. - The global denominator is the distinct SHA set across every public Tangled repository the worker has reached. The dashboard shows repository coverage next to the share and labels the figure preliminary until coverage is high. - The previous self-reported figure (32.6% on 2026-09-04) is shown as context, not silently reproduced. The share is period-scoped (30d, 90d, 1y, or all time) and is **all of your Tangled activity across all repositories** — your handles plus your bots — over that period. It is not a per-repository or per-handle ratio. GitHub is tracked separately and is not part of the Tangled denominator. The dashboard's data covers at least the last 365 days. ## Identities Personal Tangled identities are configured with a stable, validated colour: | identity | colour | |---|---| | permadeath.com | series-red (hard red) | | lance.blue | series-1 (blue) | | permadeath.games | series-3 (aqua) | | atgc.codes | series-4 (yellow) | | personal GitHub | series-6 (green) | | did.bot agents | series-5 (magenta) | | work GitHub | series-7 (violet) | The date chart and the time wheel stack by identity in these colours, or, by default, by project: the year's six busiest, owner and agents together, each keeping its colour between builds while it stays in the six, plus grey "other". A "project · identity" toggle switches both (`?by=identity`); the period pickers filter days and never recolour. The redacted work account carries no repository names. Project and recent-commit rows carry a small `T` (Tangled) or `GH` (GitHub) source badge, and Tangled project links use the fully qualified `handle/repository` form. ## Local development ```bash cargo run -- crawl --db state.sqlite3 --out dist --owned-only --skip-github cargo run -- serve --dir dist --bind 127.0.0.1:8787 ``` `--owned-only` scans only the configured personal and bot repositories, which is the fast feedback loop. Without it the worker enumerates the public Tangled index and resumes the global backfill across runs. A run is bounded by `TALLY_CRAWL_BUDGET_SECS` (default 3000). Repositories and branches are committed to SQLite as they are read, so the next run resumes exactly where this one stopped. ## Two crawl schedules The production deployment uses two tasks and two databases: - **Scheduled global (monthly)** — `crawl --global`, with `TALLY_STATE_KEY=state/global.sqlite3`. It walks every public Tangled repository, ignores the time budget, skips GitHub and the inbox, writes `TALLY_GLOBAL_SUMMARY_OUT`, uploads periodic SQLite checkpoints, and does not publish the site. It can run as N shards (below). - **Hourly owned** — `crawl --owned-only`, with the normal `state/tally.sqlite3`. It scans the personal and bot repositories, ingests recent GitHub reports, reads `TALLY_GLOBAL_SUMMARY`, publishes the site and uploads the hourly snapshot. The global summary is: ```json {"version":1,"generated_at":"2026-10-21T03:07:00Z", "repos_reached":14528,"repos_total":14528,"partial":false, "days":{"2026-10-20":1234}} ``` The global pass skips fork records, excludes commits before `TALLY_GLOBAL_IMPORTED_BEFORE` and before each repo's creation date minus `TALLY_GLOBAL_REPO_AGE_GRACE_DAYS`, and flags a repository whose default branch exceeds `TALLY_GLOBAL_REVIEW_REPO_COMMITS` for review. A per-repo page cap is the backstop for a log whose total is unavailable. Which repositories count is decided by a versioned rule set (`src/rules.rs`), recomputed from scratch on every run, so removing a rule or an exclude brings a repository back. Each repository gets an evidence vector, signals in tiers: | tier | for | against | |---|---|---| | manual | | excludes file, fork record, description says mirror/fork/backup | | strong | commits by Tangled identities (DID authors); history born on Tangled | 50+ authors with under 0.5% Tangled-identity commits; stopped at the page cap | | medium | at most 10 authors; owner keeps other repos; an original description | 10+ authors before the repo existed | | weak | complete walk; plausible size | history from before the repo existed | A manual signal excludes. A strong negative excludes, or holds the repository for review when a strong positive contradicts it; a page-cap stop holds. Otherwise a repository counts with one strong positive, two medium ones, or one medium and no medium negative. 10+ authors before the repo existed also drops its grace period. At the merge, a repository over 5% of any window (and at least 1,000 changes in it) is held unless its DID is in `global-reviewed.txt` (`TALLY_GLOBAL_REVIEWED`). Rewritten commits without a Change-Id (same author email, author time and subject) count once, and nothing dated more than a day after the run counts. The summary records `rules` (version, parameters), `rules_hash` (SHA-256 of them, the excludes and the reviewed list), the verdict tallies, `held_for_review` with what each held repository would add, `held_days`, the evidence of the largest excluded or held repositories, and `top_repos`. `crawl --global --filter-only` applies the rule set and the oversized-repo log to an existing database, rewrites the summary, and exits without walking commits. It uses indexed commit counts, so a giant repository that was never scanned is not visible to the audit until a normal run reaches it. ### Shards `TALLY_SHARD_INDEX` of `TALLY_SHARD_COUNT` (or `crawl --global --shard i/N`) crawls only the repositories whose DID hashes to that shard, under its own state key beside the single-task one: `state/global.sqlite3` becomes `state/global-shard-2-of-8.sqlite3`. A shard with no state yet starts from the single-task database, trimmed to its repositories, so moving to shards is not a re-crawl. Instead of the summary, each shard publishes its counted changes (`state/global-shard-2-of-8.counts.sqlite3`): one row per logical change with its earliest day in each count. `global-merge` (with the same `TALLY_SHARD_COUNT`, or `--counts` files) merges them and writes `TALLY_GLOBAL_SUMMARY_OUT`. A change reachable from repositories on two shards counts once, so the merge equals a single-task summary; adding per-shard daily counts would not. The merged summary is as old as its oldest shard and partial if any shard was. A repository whose history no walk has finished (a branch left unindexed, or commits none of which is marked as on the default branch) is due whatever the refresh window says, so a seeded shard walks every default branch once. A walk saves its place (the head it reads and how far into its log) after every page, and a branch is indexed only when a walk reaches its end, the import cutoff or known history; a walk cut short by a timeout, a failed page or the page cap resumes from that place next run. Until then the repository is not scanned for coverage: one stopped at the cap is a failure, the rest count in `repos_unwalked` and `coverage.repos_walk_incomplete`. Before writing a summary, the single task and `global-merge` refuse one that looks wrong: coverage under `TALLY_GLOBAL_MIN_COVERAGE_PCT`, too many repositories still to walk, or default-branch counts under `TALLY_GLOBAL_MIN_DEFAULT_RATIO` of the all-branch ones. The run fails and the published summary stays. ### Discovery and coverage The global run's universe is Tangled search, including records from before repositories had a DID (their knot serves them as `/`; the mirror doesn't), plus the repo records of accounts the relay lists and search doesn't. A sharded run finds it once: `global-discover` writes the list to `TALLY_GLOBAL_DISCOVERY` before the shards start, and each shard keeps its own repositories from it. Knots on `localhost`, private addresses and reserved names are not asked at all. A failure is tried once more after the circuit cooldown. Coverage is what the run read over what it could read. `repos_total` is the reachable repositories, `repos_fresh` those read without error this run (at or after `coverage.scanned_since`, the run's start less the refresh window), and `repos_reached` those ever read. A repository is unreachable, and outside `repos_total`, when its knot is private, or when its host or the repository itself has failed three times in a row over at least two days; anything else that fails stays in as a failure. `missing_changes_30d` estimates the logical changes of the last 30 days the unread repositories hold: each one's rate over the 30 days before its last good read, over the days since. `coverage` has the detail per shard and summed: discovered, excluded, reachable, scanned, failed, not attempted and unreachable counts, failure and unreachable reasons, how repositories were found, and the same estimate for unreachable ones. A checkpoint writes the same file with `partial:true`. The hourly run sums `days` for each selected period and hides the share when the summary is missing, stale, or partial below `TALLY_GLOBAL_MIN_COVERAGE_PCT`. A partial summary at or above that coverage is shown with its coverage beside the number. The age cutoff is `TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS` (default 9, range 1–365); use about 35 for a monthly crawl. ## Configuration | variable | default | meaning | |---|---|---| | `TALLY_PERSONAL_DIDS` | the four identities | `did\|handle\|series-N` comma separated | | `TALLY_PERSONAL_EMAILS` | empty | `address=did` aliases for older commits | | `TALLY_DIDBOT_PDS` | `https://pds.did.bot` | comma-separated didbot PDS bases | | `TALLY_TANGLED_API` | `https://api.tangled.org` | appview XRPC | | `TALLY_TANGLED_MIRROR` | `https://mirror.tangled.network` | commit/branch API | | `TALLY_GLOBAL_DISCOVERY` | unset | where `global-discover` writes the repository list and a sharded run's shards read it (`s3://…` or a path); unset or older than 36 hours, a run finds the universe itself | | `TALLY_TANGLED_RELAY` | `https://relay1.us-east.bsky.network` | relay whose `sh.tangled.repo` writers the global run lists when search lacks them; empty is off | | `TALLY_CONCURRENCY` | `8` | repositories scanned in parallel | | `TALLY_PER_HOST_RPS` | `10` | per-host request pacing | | `TALLY_CRAWL_BUDGET_SECS` | `3000` | hourly budget; `--global` ignores it | | `TALLY_GLOBAL_REFRESH_SECS` | `86400` | how often a global repo is revisited; use `2592000` for monthly | | `TALLY_OWNED_REFRESH_SECS` | `3600` | how often an owned repo is revisited | | `TALLY_GLOBAL_CONCURRENCY` | `4` | global-run repository parallelism | | `TALLY_GLOBAL_PER_HOST_RPS` | `2` | global-run per-host request pacing | | `TALLY_GLOBAL_HTTP_ATTEMPTS` | `3` | attempts per global request | | `TALLY_GLOBAL_SKIP_FORKS` | `true` | exclude fork records from the global denominator | | `TALLY_GLOBAL_SKIP_DESCRIBED_COPIES` | `true` | also exclude repos whose own description says they are a mirror, fork or backup (`[READ-ONLY] Mirror of …`, `fork of …`) | | `TALLY_BRANCH_SCOPE` | `default` | which branches count: `default`, what each repository's default branch reaches (merged work), or `all`, every branch. Both are stored and summarized, so switching needs no recrawl | | `TALLY_GLOBAL_IMPORTED_BEFORE` | `2025-01-01` | network-wide floor for imported history | | `TALLY_GLOBAL_REPO_AGE_GRACE_DAYS` | `30` | also drop commits older than each repo's Tangled creation date minus this grace | | `TALLY_GLOBAL_MAX_PAGES` | `40` | per-repo page cap; stops runaway histories | | `TALLY_GLOBAL_REVIEW_REPO_COMMITS` | `20000` | flag repos above this for manual review; not auto-excluded | | `TALLY_GLOBAL_EXCLUDES` | `global-excludes.txt` | one repository DID per line | | `TALLY_GLOBAL_REVIEWED` | `global-reviewed.txt` | held repositories the owner has reviewed and released, one DID per line | | `TALLY_GLOBAL_PROGRESS_SECS` | `600` | progress log interval | | `TALLY_GLOBAL_PROGRESS_REPOS` | `250` | progress log interval in repos | | `TALLY_GLOBAL_CHECKPOINT_SECS` | `1800` | seconds of crawling between checkpoint uploads | | `TALLY_GLOBAL_CHECKPOINT_REPOS` | `0` | also checkpoint every this many repos; 0 is off | | `TALLY_SHARD_INDEX` | `0` | this task's shard of the global crawl | | `TALLY_SHARD_COUNT` | `1` | number of global shards; 1 is the whole crawl in one task | | `TALLY_REPO_TIMEOUT_SECS` | `900` | give up on one repository after this long; 0 is no limit | | `TALLY_GLOBAL_MIN_COVERAGE_PCT` | `90` | minimum coverage for a partial share, and for publishing a global summary at all | | `TALLY_GLOBAL_MIN_DEFAULT_RATIO` | `0.4` | refuse to publish when a default-branch count falls below this share of the all-branch one; 0 is off | | `TALLY_CIRCUIT_FAILURES` | `5` | consecutive failures before a host circuit opens | | `TALLY_CIRCUIT_COOLDOWN_SECS` | `60` | host circuit cooldown | | `TALLY_GLOBAL_SUMMARY_MAX_AGE_DAYS` | `9` | 1–365; hide the share after this many days (2 nightly, 9 weekly, 16 fortnightly, ~35 monthly) | | `TALLY_GLOBAL_SUMMARY_OUT` | unset | global output: `s3://…` or local path | | `TALLY_GLOBAL_SUMMARY` | unset | hourly input: `s3://…`, `https://…` or local path | | `TALLY_MAX_REPOS_PER_RUN` | unset | optional cap for a run | | `TALLY_TIMEZONE` | unset | IANA zone for the time-of-day wheel, such as `America/New_York`; unset, each viewer sees their own zone. Converted in the browser per date, so daylight saving is right. The link preview's wheel reads hours in it too, in UTC when unset | | `TALLY_ACTIVITY_URL` | `https://vibescrobble.com` | https vibescrobble service the live "agentic activity" card polls in the browser for the first personal DID; empty turns the card off | | `TALLY_ACTIVITY_TEXT` | `true` | show each scrobble's text, model and repo in that card; `false` keeps only emoji, agent and age | | `TALLY_OG_LAYOUT` | first in `og::LAYOUTS` | the link preview's design by name: `dashboard` or `year` | | `TALLY_WORK_INBOX` | unset | `s3://…`, `https://…` or local path for one inbox payload | | `TALLY_INBOXES` | unset | comma-separated inbox payloads for multiple accounts; overrides `TALLY_WORK_INBOX` | | `GITHUB_TOKEN` | unset | personal GitHub token | | `TALLY_GITHUB_LOGIN` | token's login | personal login | | `TALLY_GITHUB_OWNERS` | `permadeath-games` | extra owners to enumerate | | `TALLY_GITHUB_REDACT_ALL` | `false` | redact every personal-account repo | | `TALLY_GITHUB_REDACT_REPOS` | empty | personal repos to redact by full name | | `GITHUB_WORK_TOKEN` | unset | optional direct work token; prefer the inbox | | `TALLY_WORK_REDACT_ALL` | `true` | redact every work repository | | `TALLY_WORK_REDACT_REPOS` | empty | comma-separated work repos to redact instead | | `TALLY_PUBLISH` | `0` | upload the site for hourly runs, or state + summary for global runs | | `TALLY_SITE_BUCKET` | unset | public site bucket | | `TALLY_STATE_BUCKET` | unset | index snapshot and inbox bucket | | `TALLY_STATE_KEY` | `state/tally.sqlite3` | index snapshot key | | `TALLY_S3_PREFIX` | empty | optional key prefix for the site | ## GitHub ### Personal account and `permadeath-games` Create a fine-grained PAT with: - **Metadata: Read-only** - **Contents: Read-only** on the repositories you want counted, including the private repositories in the `permadeath-games` organization. - Organization approval for `permadeath-games`, if the org requires it. The worker lists `/user/repos` and also enumerates every owner named in `TALLY_GITHUB_OWNERS`, so `permadeath-games` is included even when it is not returned by the authenticated account's repository list. Commits are filtered to `author=`. Store the token in SSM SecureString and give its parameter name to the infra stack as `github_token_parameter_name`. ### Either account: upload a report instead of a token The service never needs the account tokens. Run the scraper once per account on the machine that holds that token: ```bash # personal account; public repositories are named, private ones are hashed GITHUB_TOKEN=... python3 scripts/github_activity.py \ --label personal --login \ --s3 s3://tally-state-ACCOUNT/inbox/personal-activity.json # work account; every name is hashed, including public repositories GITHUB_WORK_TOKEN=... \ python3 scripts/github_activity.py --label work \ --s3 s3://tally-state-ACCOUNT/inbox/work-activity.json ``` Private and internal repositories are hashed by default, including when `--no-redact-repos` is passed. Public repository names are used unless `--redact-all` is passed. A `--label work` report hashes **every** repository, public or private, because a public repository name can still disclose what the work account was doing. `--reveal-private-repos` names private repositories on a personal report and should only be used for repositories you are willing to publish. The repo-hash salt is created on first use at `~/.config/tally/repo-salt` with mode `0600`, or at `$TALLY_SALT_FILE` / `--salt-file` if set. No salt argument is needed. Keep that file stable and private: changing it changes every repository hash. `--salt` and `WORK_REPO_SALT` still override it for one-off runs. On the first run each collects 365 days across every branch; later runs re-read two days of overlap and the service deduplicates by SHA/Change-Id. The state file is per label, so the two runs do not overwrite each other. It also records each repository's `pushed_at`, so unchanged repositories are skipped without listing their branches; `--full` ignores that cache. Each commit is marked with whether the repository's default branch has it, and the newest report's mark replaces an older one. Reports from before the mark existed count every commit; run each label once with `--full` so history uploaded earlier is marked too. `--upload-url` sends the payload as a plain HTTPS `PUT`, with no AWS CLI involved. The bearer token is read from `TALLY_UPLOAD_TOKEN`, then `TALLY_UPLOAD_TOKEN_FILE`, then `~/.config/tally/upload-token`. That is the hook for a stable ingest endpoint; a presigned S3 URL also works but expires and has to be regenerated. Point the service at both payloads: ```bash TALLY_INBOXES=s3://tally-state-ACCOUNT/inbox/personal-activity.json,s3://tally-state-ACCOUNT/inbox/work-activity.json ``` The payload contains only: ```json {"version":1,"label":"personal","redacted":false, "generated_at":"…","commits":[ {"sha":"<40 or 64 hex>","authored_at":"2026-10-19T15:00:00Z", "repo":"owner/public-repo","redacted":false,"change_id":"I…"}, {"sha":"<40 or 64 hex>","authored_at":"2026-10-19T16:00:00Z", "repo":"h_ab12…","redacted":true,"change_id":""} ]} ``` Each commit carries its own `redacted` flag, so one payload can mix named public repositories with hashed private ones. The top-level `redacted` is true only when every repository is hashed; older payloads may omit the per-commit flag and fall back to it. For `--label work`, `repo` is normally a salted `h_ab12…` hash. There is no field for a login, commit message, URL, author name or email. `change_id` is only the Gerrit `Change-Id:` trailer value, so rebases collapse to one logical change just like atgc commits. The service reads only the fields above and ignores extras. The work machine's credential needs one permission: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": ["arn:aws:s3:::tally-state-ACCOUNT/inbox/work-activity.json"] }] } ``` The infrastructure repo owns the state bucket and the write-only policy. Point the scraper at the `s3://…/inbox/work-activity.json` URI it provides. A direct `GITHUB_WORK_TOKEN` is also supported, but it requires the service to hold the token and see repository names. Prefer the inbox. ## Deployment This repository is the application only. AWS infrastructure lives in the private platform Terraform repo (`envs/devinfra/site*.tf` and `modules/site`). See [docs/GLOBAL-RERUN.md](docs/GLOBAL-RERUN.md) for the rebuild and global re-run contract. The app side of the contract is: - build the image from `Dockerfile` and push it to the platform ECR repo - run the nightly task as `crawl --global`, with its own `TALLY_STATE_KEY=state/global.sqlite3`, `TALLY_GLOBAL_SUMMARY_OUT`, no time budget, and no GitHub/inbox configuration - run the hourly task as `crawl --owned-only`, with `TALLY_STATE_KEY` pointing at the hourly snapshot and `TALLY_GLOBAL_SUMMARY` pointing at the nightly summary - give the task roles the S3 state/site permissions, SSM token access and networking described in the infra repo The worker is a one-shot Fargate task. CloudFront serves the static site; there is no load balancer or always-on service. ## Tests ```bash cargo test cargo check --features aws cargo fmt --check cargo clippy --all-targets --features aws -- -D warnings ``` The unit tests cover the de-duplication invariant (same SHA on many branches and in two repositories), identity upgrades without duplication, the Git SHA validator, repository-hash sanitising, the palette's stable bot colours, the S3 prefix/object-key rules and the global-summary period maths. ## Limits - Private Tangled repositories are not part of the public index and are not counted. - The global share is only meaningful once the repository coverage shown next to it is high. Until then the site labels it preliminary. - A work repository is only as hidden as its hash and the local salt. The hash prevents the public site from naming it; it is not encryption. - GitHub commits are filtered to the authenticated login. Co-authored commits are counted under their primary author.