ordner
replace big-tech and host basic infrastructure yourself! ordner.fyi
Star 0
Fork 0
Watch

Ansible recipes for srv.ordner.fyi — knot, spindle, infra backups, srv routing #19

Merged pipobscure.com
targeting main
Merged opened by and assigned to pipobscure.com
targeting main

What this is #

The Ansible recipe that stands up srv.ordner.fyi (Hetzner Falkenstein, Debian 13, 64 GB RAM, 2×512 GB SSD) as the new ordner host — written against the migration guide (research/srv-migration/host-migration-guide.md, draft v1) and the lab's actual, verified configuration. It is deliberately the same artifact crash-recovery would run: provision a box, restore from the bucket, re-join, done. This PR ships recipes only — nothing has been executed against srv (no access yet).

The one hard rule is baked in everywhere it can be mistaken: the suite hostname stays lab.ordner.fyi. The machine is srv; the install is lab.ordner.fyi, before and after the cutover (passkeys/rpId, OIDC + atproto client ids, the TLS lineage paths all derive from the kept name).

The files #

deploy/ansible/
  site-srv.yml                  the srv play: provisioning + verify
  render-check-srv.yml          render everything locally, touch no host
  RUNBOOK-srv.md                per-tag runbook, pre/post operator lists, gaps
  group_vars/srv.yml            the [srv] group's settings (no credentials)
  inventory.srv.example.ini
  files/infra-crypt-env.py      the one HKDF derivation, three consumers
  files/prove-ptr.py            §9.2 precheck (runs as a pre_task, tagged always)
  files/prove-egress.py         §9.7 outbound 25/465 check (verify play)
  roles/knot/                   new
  roles/spindle/                new
  roles/backup/                 new
  roles/nginx-srv/              new

site-srv.yml orders bundle → node → base → seaweedfs → resolver → deploy → knot → spindle → backup → nginx-srv, with the PTR/forward-DNS precheck and the Debian-13 refusal as always-tagged pre_tasks, and a verify play (--tags verify): the infrastructure checks run any time (knot health + owner DID, spindle /_health, the podman socket, TLS routing per name, outbound 25/465, backup timer), the suite's /status + certificate horizon only with -e wizard_done=true — they are meaningless before the wizard.

The new roles #

knot — knot2 is the Rust crate knot-server in the tangled-core monorepo (the guide/task's "build with Go" fits the old Go knotserver, not knot2; see gaps). Install modes: controller (default — the lab's actual path: built once, copied), release (URL var), source (clone + cargo build --release -p knot-server, toolchain 1.96.0 pinned). The config template uses the monorepo's own knot2/example.toml keys — hostname knot.ordner.fyi, admin DID did:plc:rwypz6edxmzyhob2k3rsl2fw, loopback listeners 5555/2222, object_format = "sha1" (CI-spindle compat), scan_path/sealed_key_file/master_key_env per the lab's layout. The unit adapts the monorepo's shipped systemd/knot.service hardening. The state restore pulls knot/etc + knot/state from the backup folder through the crypt remote; on a fresh install it renders the config, generates an SSH host key and mints a new KNOT_MASTER_KEY (loudly: that is a new knot, not the recovery path).

spindle — go build ./cmd/spindle with pinned Go 1.26.9 (checksum from go.dev/dl; Debian 13's golang is below go.mod's go 1.26), git 2.56.0 built to /opt/git-2.56.0 (checksum pinned; PATH override only inside the unit), rootless podman (subuids, enable-linger, the user's podman.socket), and the verified pasta fix (pasta_options = ["-a", "10.0.2.100"], array form — the map form fails to decode). The env file renders the binary's real variable surface, read from spindle/config/config.go: SPINDLE_SERVER_{HOSTNAME,OWNER,LISTEN_ADDR, DB_PATH,REPO_DIR,LOG_DIR,DOCKER_SOCKET}, SPINDLE_NIXERY_PIPELINES_* (nixery.dev per the operator's direction; lab limits: 2 pipelines / 4 GiB / 15 m). Same crypt-remote restore; a restored env is authoritative and its hostname/owner are drift-checked against group_vars.

backup — svc-infra-backup.timer (daily 03:40 + jitter, Persistent=). The script reads the master key and the blobS3 coordinates out of svc.json (never ships them), derives the crypt passphrase (HKDF-SHA-256, salt 0³², info svc.backup.v1, hex), and syncs through the crypt remote into backups/infrastructure/<host>/: the knot state, the spindle state (SQLite via VACUUM INTO snapshots — never live db copies; knot2 keeps no db, the loop is generic), /srv/www, and the non-secret configs (nginx, units, letsencrypt — which keeps the DANE-pinned lineage). Not svc.json (the phrase stays the root; the bucket must never hold a key derived from itself) and not /opt/svc/state (the high-water journals never travel). Retention: mirror + --backup-dir dated archive, --min-age cleanup (30 d, var). The folder sits outside the install prefix, so the doctor's gc never sweeps it. Pre-wizard runs exit 0 with a notice and the timer keeps ticking.

nginx-srv — the srv variant of roles/nginx. Port-80 vhost for every name the box answers; both lineages pre-issued via DNS-01 through the Hover hook pattern from the lab (wildcard ordner.fyi + *.ordner.fyi, and lab.ordner.fyi with --reuse-key — DNS-01 because pre-cutover an HTTP-01 challenge served by srv is validated against the lab; pre-issuing is the guide §6 option that lets the wizard write the mail TLS section and lets §5's verification windows exercise SMTP before the flip); the renewal hooks 10/20/30; the knot vhost (git over HTTPS: client_max_body_size 0, hour-long timeouts), the spindle vhost, the static catch-all (root /srv/www/$host, /bundles/, www→apex), and the suite vhost installed from the same reference config the stock role ships — the lab's svc.conf content, byte for byte, with the kept hostname substituted.

What a real run would do #

site-srv.yml (restore case: paste the master key at the prompt + vault the infra_s3_* and hover_env inputs):

  1. prove the PTR forward-confirms (refuses otherwise; prints the srv-vs-lab naming note — expected pre-cutover);
  2. build + push bundles, install Node 26.11.1, base packages + ClamAV, weed 4.47 trio on loopback, unbound as the system resolver, deploy the suite stopped-in-wait (ten units, five reclamation timers);
  3. install knot + spindle (restore their states when prompted), start both, wait on their health routes;
  4. arm the backup timer (first real backup lands after the wizard);
  5. issue both TLS lineages via Hover DNS-01, install the routing table, nginx -t, reload;
  6. wait for admin and print the wizard URL.

Then the runbook's operator sequence: --tags verify (infra half) → the storage phase (guide §4 — operator choice, not playbook work) → the wizard with the lab's values → systemctl start svc-infra-backup.service → the cutover (guide §6) → --tags verify -e wizard_done=true.

What stays manual (deliberately) #

  • the setup wizard — the designed join path; the playbook must never hold the bucket credentials or the master key;
  • the storage phase (cluster join vs endpoint consolidation — §4): the playbook stands a standalone loopback weed trio and makes no cross-DC port or topology decision;
  • the DNS flips and the cutover freeze itself — sequential, one live node at a time (§9.6); nothing here touches the lab;
  • the PTR/mail-name decision at the cutover (set the PTR to the kept name or switch the MX — §6.6) and the rpId re-bind decision (§0) before the wizard;
  • the Hetzner 25/465 unblock is account-level; the recipe only verifies it (prove-egress.py, hard-failing in the verify play);
  • the disk layout (RAID1/LVM/swap — §2) from the rescue system, before the first run.

Gaps found in the guide's assumptions #

  1. "the recipe may build knot2 with Go" — knot2 is Rust; the Go knotserver in the same monorepo is the old knot. The recipe's modes cover reality (controller copy / release / cargo source build).
  2. The crypt passphrase input — the guide says a recovery re-derives the passphrase "from the phrase", but the phrase→masterKey derivation lives inside the wizard. The recipe derives from the masterKey value in svc.json on both the backup and the restore path, so the operator must export that value from the lab before it goes away. If the phrase itself is to be the root, the derivation has to move into the wizard's algorithm first.
  3. The lab's spindle env carried a DOCKER_HOST entry (inventory open item) the binary does not read: the real key is SPINDLE_SERVER_DOCKER_SOCKET (a socket path, bind-mounted into job containers, exposed inside as DOCKER_HOST=unix://<path>).
  4. example.toml confirms more required knot2 fields than the lab notes recorded (ssh_host_key_file, plc_directory) — the template covers them; the restore's drift check fails the run if the lab's live config disagrees, so nothing is guessed silently.
  5. The wizard's rpId editability (§0) and the DKIM selector (§8) remain unresolved guide items — outside playbook scope, but the cutover checklist waits on both.

Verification performed #

  • ansible-playbook --syntax-check on all four plays (site.yml and render-check.yml untouched and still green).
  • render-check-srv.yml renders knot/spindle units + configs, backup unit/timer/defaults and all four srv vhosts against group_vars; systemd-analyze verify + calendar (backup timer elapses shown).
  • bash -n/sh -n on the three shell scripts, py_compile on the three python files, and a functional test of the derivation helper: HKDF determinism + key separation, and both input modes (svc-json / master-key stdin) emitting the correct hetzner + backupcrypt remotes.
  • Config key names and env var names transcribed from the tangled-core monorepo source (knot2/example.toml, spindle/config/config.go, spindle/engines/nixery/engine.go), not from memory.

This pull request has no versions submitted yet.