# Deployment sovrn runs on NixOS. Every host is deployed with Colmena from the fleet repository (`~/projects/servers`); this repository exports the NixOS modules and packages the fleet puts on hosts. Deploys run from the operator's machine (no CI for now). The migration that got here (Ansible → NixOS + Colmena, bug 3beadb2) is recorded in the fleet's `docs/fleet-migration-plan.md`. ## Why this shape Priorities, in order: simplicity, extensibility, disaster recovery, security, monitoring, upgrades. The deciding constraint is operational simplicity under network partitions: shared-store clusters turn every host failure into a consensus problem. Cells avoid that: every database is an embedded SQLite file with continuous off-box replication, so a dead host is replaced, not failed over at the storage layer. The first deployment used Debian + Ansible for transparency. It moved to NixOS + Colmena (2026-10) for declarative, reproducible hosts that several projects can share, with VM tests for every role and one place (the fleet) that deploys everything. ## Hosts and roles The fleet's `hosts.json` lists each host's roles; sovrn provides these (`nixosModules` in `flake.nix`): | Role | Module | Where | Notes | |---|---|---|---| | `sovrn-cell` | `cell` | one Hetzner box per cell, `mxN..sovrn.at` | Exclusive: nothing else on the box but its region's `sovrn-pds` | | `sovrn-metrics` | `metrics` | the shared box (`mxb.eu.sovrn.at`, netcup) as `infra.sovrn.at` | VictoriaMetrics, VictoriaLogs, vmalert, Alertmanager | | `sovrn-relay` | `relay` | the shared box as `mxb.eu.sovrn.at` | The pri-20 backup MX for every cell | | `sovrn-pds` | `pds` | the region's cell and the shared box | A node of the region's vlpds cluster (`settings.sovrnPds`); allowed beside `sovrn-cell` | | (none) | `telemetryEdge` | any host without a sovrn role | The cell, metrics and relay roles bring their own edge | Fleet-wide settings (`nix/fleet.nix`, overridable per host from the inventory's `settings.sovrn`) include the landing cell (`rootHost`, the one cell serving `sovrn.at`; the fleet asserts it names a cell), the backup MX, the outbound relay (SMTP2GO), the object store per region (`objectStores`: EU on Scaleway Warsaw, US on R2's US jurisdiction) and ACME contacts. ## Cells Each cell is independent (no coordination between cells) and holds a handful of tenants. Growth means another cell. Services on a cell: - **Stalwart** 0.16 (nixpkgs, version-pinned in `nix/overlay.nix`): mail, SQLite store (`/var/lib/stalwart/stalwart.db`); message blobs in the cell's bucket under `stalwart-blobs/` (`nix/stalwart/blobstore.plan.json`). Configured by the checked-in apply plan (`nix/stalwart/*.plan.json`), converged in recovery mode on every start. Its web UI is disabled. - **sovrnd**: control plane and app view (`sovrn.db`, `oauth.db`, `waitlist.db` on the landing cell). - **PDS** (role `sovrn-pds`): a node of the region's vlpds cluster (`pds..sovrn.at`), with a standby node on the shared box, over a WireGuard link; all its state lives in the region's PDS bucket. sovrnd serves each hosted domain's handles on it at runtime. See `nix/README.md` ("The PDS") and ADR-0009's amendment. - **Caddy**: TLS edge (the cell hostname, the PDS name and on-demand TLS for handle hosts, the ACME HTTP-01 challenge proxy for Stalwart; `sovrn.at` on the landing cell). - **Backups** and **restore** (`nix/modules/backup.nix`), below. - **Telemetry edge**: node_exporter, vmagent, Vector, pushing to `https://infra.sovrn.at` with basic auth. **Mail flow:** inbound MX 10 is the cell; MX 20 is the backup relay `mxb.eu.sovrn.at`, which queues for up to 7 days and re-delivers to the cell's relay ingress (port 2525, firewalled to the relays' addresses, no SPF/DMARC re-check). The relay learns each cell's domains by polling the cell's `GET /backup/domains` every minute (`sovrn-route-sync`) and keeps the last-known list while a cell is down. Its retry backoff grows to 15 minutes. Outbound goes through SMTP2GO, which DKIM-signs (branded CNAMEs, bug e76f226). **Failover: recycled primary IPs, never DNS.** Each cell's primary IPv4 and IPv6 are Hetzner Primary IPs that outlive any server. A rebuilt cell gets the same name, the same IPs (moved from the powered-off old server) and the same secrets, so DNS, PTR, MX and the relay's settings never change. See `docs/runbooks/recovery-primary-ip.md`. **Backups:** one bucket per cell, `-` (`eu-mx99`), in its region's object store (`objectStores.` in `nix/fleet.nix`). Inside: `litestream//` (Stalwart, sovrnd; daily snapshots kept 3 days), `stalwart-blobs/` (Stalwart's message blobs, written by Stalwart itself), `caddy-storage/` (certificates). The PDS needs no backup job: vlpds keeps everything in its own bucket (`-pds`, prefix `vlpds/`). Litestream replicates continuously; a 5-minute probe proves every database caught up in the bucket and refreshes `litestream.ok`; a 15-minute rclone run refreshes `rclone.ok`. A guard arms backups only for a cell whose bucket is empty, so a rebuilt cell never overwrites its own backups. **Restore is automatic.** `sovrn-restore` runs before Stalwart, sovrnd and Caddy on every boot: with no local databases and a non-empty bucket it restores everything (integrity-checked), then arms backups; with an empty bucket it arms a new cell; with local data it does nothing. **Health:** sovrnd's unauthenticated `GET /healthz` aggregates Stalwart, the local vlpds node, DNS, SQLite writability, disk and backup freshness (15 m / 45 m max ages). 200 when healthy, 503 with the same JSON naming each failure. Monitor with Larm: `GET https:///healthz`, expect 200. ## The metrics box `infra.sovrn.at` (on the shared box) runs VictoriaMetrics (90 days), VictoriaLogs (30 days), vmalert (metrics and log rules), Alertmanager (mail via Resend to `bt@rtw.run`) and a public `/healthz`. Everything listens on loopback; Caddy is the edge with basic auth (user `cell`, the shared `metrics-password`) on everything but `/healthz`. UIs: `/vmui/`, `/select/vmui/`, `/vmalert/`, `/vmalert-logs/`, `/am/`. Labels: every series and log line carries `cell` (who owns it: the cell's hostname, or on the shared box `infra.sovrn.at`, `mxb.eu.sovrn.at`, `moods`, `servers`), `host` (where it runs) and `service`; logs carry `level`. Alert rules group by `cell`. Staging cells judge their disk trend over 6 h (they also build software). Dashboards, alert responses and staged-failure checks: `docs/runbooks/metrics-box.md`. ## Operations Everything below runs in the fleet repository (`cd ../servers`) from any shell (its recipes call colmena and nixos-anywhere through its dev shell). This repository's `just` host recipes forward there (`just fleet` prints the layout). | Task | Command | |---|---| | Deploy sovrn's last commit everywhere it runs | `just deploy-project sovrn` (then commit `flake.lock`) | | Deploy one host | `just deploy ` | | Preview a deploy | `just diff-host `; `just deploy dry-activate` | | New cell | `docs/runbooks/provision-cell.md` | | Per-host secrets | `just gen-host-secrets ` (never rotates; `DRY=1` previews) | | Check secrets | `just check-secrets [fqdn]` | | Rebuild a cell | `docs/runbooks/recovery-primary-ip.md` | **Commit before deploying.** The fleet builds sovrn's last commit (in jj, `@-`), never the working copy; `just deploy` here refuses uncommitted UI assets for that reason. **Secrets** live in the age-encrypted `secrets` store (`~/projects/data`): `sovrn/shared/` and `sovrn/hosts//`. Colmena decrypts them on the deploying machine and uploads them to `/var/lib/sovrn-keys`; nothing secret enters the Nix store or a repository. Values sovrn can generate come from `apps.gen-secret` (called by the fleet's `gen-host-secrets`); provider keys (object store, Resend, SMTP2GO) are stored by hand. **arm64 cells.** Cells may be Hetzner CAX (aarch64). The deploying machine builds aarch64 under qemu emulation or hands builds to the staging cell `mx99` as a Nix remote builder (`/etc/nix/machines`); production cells reuse the same store paths, so they never build anything. Only sovrn's Go commands are built; everything else comes from cache.nixos.org, and the vlpds image is fetched by digest on the deploying machine and shipped by Colmena. **Upgrades.** nixpkgs is pinned in `flake.lock` and shared with moods and the fleet: `just update-nixpkgs` here and in moods, then `just sync-nixpkgs` in the fleet. Stalwart's version is asserted in `nix/overlay.nix`, so a nixpkgs bump that moves it fails evaluation until the pin is changed on purpose. ## Verification - `nix build .#checks.x86_64-linux.`: VM tests for every role and module: `cell`, `stalwart`, `stalwart-plan`, `metrics`, `relay` (queue and drain), `telemetry`, `backup`, `restore`. - Staging first: deploy to `mx99`, test live (a test domain, mail in and out, the relay drain), then the production cells. - Recovery is rehearsed on `mx99` (fleet plan, Phase 10).