walknot
A knot based on walgit / s3 compare and swap that supports HA and Kubernetes
Star 4
Fork 0
Watch
29 1 1
Files
.tangled
ci
crates
deploy
docs
interop
lexicons
third_party
_typos.toml
.dockerignore
.gitignore
Cargo.lock
Cargo.toml
Containerfile
example.toml
justfile
LICENSE
README.md
rust-toolchain.toml
UPSTREAM-README.md
UPSTREAM.md
README.md

walknot #

disclaimer: slop

A self-hostable Tangled knot whose storage is an S3 bucket.

walknot is knot2 — Tangled's git

  • atproto knot — with its storage rebuilt on the design of walgit: every repository is a write-ahead log in an object store, and every pod's disk is a disposable cache. Run one replica or many (Kubernetes, with Redis); they are interchangeable, and losing any of them — or all their disks — loses nothing.

It speaks exactly knot2's protocol (XRPC, smart HTTP and SSH git, subscribeRepos, records and COBs), so the Tangled appview, knotmirror, hydrant and spindle work with it unchanged. Without a bucket configured it is knot2.

How it works #

  • Repositories are WALs. repos/<shard>/manifest.json is rewritten with compare-and-swap (If-Match) on every commit — the only consensus there is. Ref edits are immutable log entries; objects are content-addressed packs; checkpoints fold the log. Lost responses are resolved by re-reading, so every acknowledged write is durable exactly once. → crates/knot-wal, crates/knot-store
  • Local directories are caches. Opening a repository revalidates it with one conditional GET and folds in what other replicas committed; writes go through the WAL first and are applied locally after. → crates/knot-git/src/storage.rs
  • One writer per repository at a time, by a lease that moves to whichever replica needs to write (re-entrant, renewed while used, released when idle). The CAS still guards correctness if a lease is lost.
  • Knot-wide state (members, registry, outbox, sidecars, LFS, sealed keys, SSH host key) lives in the bucket too; a leader lease elects the replica that runs singleton jobs.
  • Redis (only with several replicas) sequences the firehose knot-wide and shares single-use token nonces. It holds nothing durable.

docs/DESIGN.md has the details; UPSTREAM.md how this tree maps to Tangled's.

Running it #

# config.toml — the knot2 settings you already know, plus:
[storage]
backend = "s3"
s3_bucket = "my-knot"
# s3_endpoint = "https://minio.example.com"   # any S3-compatible store with conditional writes

[redis]
url = "redis://redis:6379"                     # required for more than one replica
  • Credentials come from the usual AWS_* environment (or the pod's cloud identity).
  • KNOT_MASTER_KEY must be the same on every replica: it seals the knot's identity, which lives in the bucket.
  • At boot the knot verifies the store really enforces If-None-Match/If-Match and refuses to start otherwise.

Kubernetes: deploy/kubernetes/base (Deployment, Services, PodDisruptionBudget); deploy/kubernetes/overlays/dev adds MinIO and Redis for a local cluster.

Image: docker build -t walknot . (--build-arg PROFILE=container for a faster build).

Migrating an existing knot2: stop it, set storage.backend = "s3", then knot-server import --repos <old scan_path> --lfs <old lfs store_path> publishes every repository (the meta repo included) and LFS object. The old directory is copied, never modified. Keep the same KNOT_MASTER_KEY and secrets.sealed_key_file / SSH host key files for the first boot: they are sealed into the bucket then.

Testing #

cargo test --workspace                       # knot2's suite, local storage
KNOT_TEST_STORAGE=wal cargo test --workspace # the same suite, every repository in a bucket
ci/test-linux.sh                             # in a Linux container, as CI does

With real services (docker run quay.io/minio/minio server /data, redis):

export WALKNOT_TEST_S3_BUCKET=… WALKNOT_TEST_S3_ENDPOINT=http://127.0.0.1:9000 \
       AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… WALKNOT_TEST_REDIS=redis://127.0.0.1:6379
cargo test -p knot-store -p knot-wal -p knot-redis    # contract, protocol, chaos on real S3
cargo test -p knot-server --test replicas_e2e         # three real replicas
ci/kind-smoke.sh                                      # 3 replicas on a kind cluster

What is covered: the store contract (CAS, create-if-absent, conditional reads) on MemoryStore and MinIO; the WAL protocol under every injected fault, seeded chaos, racing writers and a proptest model; leases; materialization checked by git fsck; maintenance and compaction racing other writers; multi-replica behaviour in-process (pushes, races, firehose order, tickets, ACL changes, token replay) and with real processes; and knot2's own suite in both storage modes.

Not yet #

  • Repositories larger than a pod's cache (walgit's remote range reader).
  • Pruning unreachable git objects (repacks keep everything). Retired packs, superseded sidecars and orphaned log entries are garbage-collected.
  • Firehose replay from a cursor scans every repository's chain (fine for small knots).
  • Multi-pack index and bitmaps are not built for bucket-backed repositories.