Goal #

While we say "Knot is just an atproto PDS that use git as a storage", there is way more stuffs going inside its internal storage. Most of them are not exposed to public endpoints and just make read/write slower. Atproto already technically has a commit history from its spec, so we don't need to make sidecar history as atproto history is everything clients will see and understand. This and following PR will simplify the knot's internal atproto repo storage.

Model comparison #

Before: #

Three git refs holding atproto repository state:
- refs/cobs/<nsid>/<origin-oid> - holding internal per-record history
- refs/cob-checkpoints/<nsid>/<origin-oid> - holding folded state of each records
- refs/atproto/commit - holding settled atproto repository.
Every COB records have their own change history under refs/cobs/<collection>/<origin-oid> refs, pointing a tip commit. These commits are signed individually with secp256k1 and hold the change actor. tree can hold two of three entries: payload and record or tombstone.
refs/cob-checkpoints/* points to a snapshot of folded state at change tip.
refs/atproto/commit points to a commit. One commit per atproto commit. Tree holds atproto commit data + optional firehose event frame. Each commit's blocks/ is the parent's plus the new blocks. Old MST nodes and old record versions stay in the tree, so blocks/ grows forever.

refs/cobs/<nsid>/<origin-oid> -> commit
commit
  cob-type   <nsid>
  cob-author <knot did:key>
  cob-sig    <hex secp256k1 sig>
  tree
    payload               -> blob: dag-cbor ***change*** payload
    record (or tombstone) -> blob: exact atproto record bytes

refs/cob-checkpoints/<nsid>/<origin-oid> -> blob: folded record state

refs/atproto/commit -> commit
commit
  tree
    blocks/<c7>/<c8>/<cid> -> blob: MST node or record body
    commit                 -> blob: signed atproto commit
    frame                  -> blob: the firehose event frame for this commit (optional)

We have three different refs with three different commit shape.

After: #

One git ref refs/atproto/commit holding source of truth. It points to tip of git commits where each commit matches with atproto commit.

refs/atproto/commit -> commit
commit
  tree
    COMMIT                        -> blob: signed atproto commit
    records/<nsid>/<hash2>/<rkey> -> blob: record body (hash2 is `sha256(rkey)[0:2]`)

Way less git data to write on each operations.
No per-record history. atproto commit log is all knot has.
No internal "change" payload. change is determined by diffing two versions (that's how spindle is doing today to detect PR changes.)
Per-record history walk can be done with help of bloom filters.
It is objectively slower, but can be improved a lot by indexing heavily.

Deploy plan #

phase 1:

deploy knot1 with this PR merged. run knot migrate warm-up to prepare the migration for 10k repos (this PR)

phase 2:

prepare v1.21 knot; legacy store completely swapped with new one (next PR)
prepare second foster knot with data seeded as new model
deploy knot1 and knotmirror with new store. knot1's foster knot configured to one prepared in step 2.
swap prod foster knot to new one.