sovrn
Experimental AT protocol email hosting
Star 0
Fork 0
Watch
sovrn docs deployment.md
9.1 kB · 165 lines
Markdown
at main

Deployment #

sovrn runs on NixOS. Every host is deployed with Colmena from the fleet repository (~/projects/servers); this repository exports the NixOS modules and packages the fleet puts on hosts. Deploys run from the operator's machine (no CI for now). The migration that got here (Ansible → NixOS + Colmena, bug 3beadb2) is recorded in the fleet's docs/fleet-migration-plan.md.

Why this shape #

Priorities, in order: simplicity, extensibility, disaster recovery, security, monitoring, upgrades. The deciding constraint is operational simplicity under network partitions: shared-store clusters turn every host failure into a consensus problem. Cells avoid that: every database is an embedded SQLite file with continuous off-box replication, so a dead host is replaced, not failed over at the storage layer.

The first deployment used Debian + Ansible for transparency. It moved to NixOS + Colmena (2026-10) for declarative, reproducible hosts that several projects can share, with VM tests for every role and one place (the fleet) that deploys everything.

Hosts and roles #

The fleet's hosts.json lists each host's roles; sovrn provides these (nixosModules in flake.nix):

Role Module Where Notes
sovrn-cell cell one Hetzner box per cell, mxN.<region>.sovrn.at Exclusive: nothing else on the box but its region's sovrn-pds
sovrn-metrics metrics the shared box (mxb.eu.sovrn.at, netcup) as infra.sovrn.at VictoriaMetrics, VictoriaLogs, vmalert, Alertmanager
sovrn-relay relay the shared box as mxb.eu.sovrn.at The pri-20 backup MX for every cell
sovrn-pds pds the region's cell and the shared box A node of the region's vlpds cluster (settings.sovrnPds); allowed beside sovrn-cell
(none) telemetryEdge any host without a sovrn role The cell, metrics and relay roles bring their own edge

Fleet-wide settings (nix/fleet.nix, overridable per host from the inventory's settings.sovrn) include the landing cell (rootHost, the one cell serving sovrn.at; the fleet asserts it names a cell), the backup MX, the outbound relay (SMTP2GO), the object store per region (objectStores: EU on Scaleway Warsaw, US on R2's US jurisdiction) and ACME contacts.

Cells #

Each cell is independent (no coordination between cells) and holds a handful of tenants. Growth means another cell.

Services on a cell:

  • Stalwart 0.16 (nixpkgs, version-pinned in nix/overlay.nix): mail, SQLite store (/var/lib/stalwart/stalwart.db); message blobs in the cell's bucket under stalwart-blobs/ (nix/stalwart/blobstore.plan.json). Configured by the checked-in apply plan (nix/stalwart/*.plan.json), converged in recovery mode on every start. Its web UI is disabled.
  • sovrnd: control plane and app view (sovrn.db, oauth.db, waitlist.db on the landing cell).
  • PDS (role sovrn-pds): a node of the region's vlpds cluster (pds.<region>.sovrn.at), with a standby node on the shared box, over a WireGuard link; all its state lives in the region's PDS bucket. sovrnd serves each hosted domain's handles on it at runtime. See nix/README.md ("The PDS") and ADR-0009's amendment.
  • Caddy: TLS edge (the cell hostname, the PDS name and on-demand TLS for handle hosts, the ACME HTTP-01 challenge proxy for Stalwart; sovrn.at on the landing cell).
  • Backups and restore (nix/modules/backup.nix), below.
  • Telemetry edge: node_exporter, vmagent, Vector, pushing to https://infra.sovrn.at with basic auth.

Mail flow: inbound MX 10 is the cell; MX 20 is the backup relay mxb.eu.sovrn.at, which queues for up to 7 days and re-delivers to the cell's relay ingress (port 2525, firewalled to the relays' addresses, no SPF/DMARC re-check). The relay learns each cell's domains by polling the cell's GET /backup/domains every minute (sovrn-route-sync) and keeps the last-known list while a cell is down. Its retry backoff grows to 15 minutes. Outbound goes through SMTP2GO, which DKIM-signs (branded CNAMEs, bug e76f226).

Failover: recycled primary IPs, never DNS. Each cell's primary IPv4 and IPv6 are Hetzner Primary IPs that outlive any server. A rebuilt cell gets the same name, the same IPs (moved from the powered-off old server) and the same secrets, so DNS, PTR, MX and the relay's settings never change. See docs/runbooks/recovery-primary-ip.md.

Backups: one bucket per cell, <region>-<cell> (eu-mx99), in its region's object store (objectStores.<region> in nix/fleet.nix). Inside: litestream/<db>/ (Stalwart, sovrnd; daily snapshots kept 3 days), stalwart-blobs/ (Stalwart's message blobs, written by Stalwart itself), caddy-storage/ (certificates). The PDS needs no backup job: vlpds keeps everything in its own bucket (<region>-pds, prefix vlpds/). Litestream replicates continuously; a 5-minute probe proves every database caught up in the bucket and refreshes litestream.ok; a 15-minute rclone run refreshes rclone.ok. A guard arms backups only for a cell whose bucket is empty, so a rebuilt cell never overwrites its own backups.

Restore is automatic. sovrn-restore runs before Stalwart, sovrnd and Caddy on every boot: with no local databases and a non-empty bucket it restores everything (integrity-checked), then arms backups; with an empty bucket it arms a new cell; with local data it does nothing.

Health: sovrnd's unauthenticated GET /healthz aggregates Stalwart, the local vlpds node, DNS, SQLite writability, disk and backup freshness (15 m / 45 m max ages). 200 when healthy, 503 with the same JSON naming each failure. Monitor with Larm: GET https://<cell>/healthz, expect 200.

The metrics box #

infra.sovrn.at (on the shared box) runs VictoriaMetrics (90 days), VictoriaLogs (30 days), vmalert (metrics and log rules), Alertmanager (mail via Resend to bt@rtw.run) and a public /healthz. Everything listens on loopback; Caddy is the edge with basic auth (user cell, the shared metrics-password) on everything but /healthz. UIs: /vmui/, /select/vmui/, /vmalert/, /vmalert-logs/, /am/.

Labels: every series and log line carries cell (who owns it: the cell's hostname, or on the shared box infra.sovrn.at, mxb.eu.sovrn.at, moods, servers), host (where it runs) and service; logs carry level. Alert rules group by cell. Staging cells judge their disk trend over 6 h (they also build software). Dashboards, alert responses and staged-failure checks: docs/runbooks/metrics-box.md.

Operations #

Everything below runs in the fleet repository (cd ../servers) from any shell (its recipes call colmena and nixos-anywhere through its dev shell). This repository's just host recipes forward there (just fleet prints the layout).

Task Command
Deploy sovrn's last commit everywhere it runs just deploy-project sovrn (then commit flake.lock)
Deploy one host just deploy <fqdn>
Preview a deploy just diff-host <fqdn>; just deploy <fqdn> dry-activate
New cell docs/runbooks/provision-cell.md
Per-host secrets just gen-host-secrets <fqdn> (never rotates; DRY=1 previews)
Check secrets just check-secrets [fqdn]
Rebuild a cell docs/runbooks/recovery-primary-ip.md

Commit before deploying. The fleet builds sovrn's last commit (in jj, @-), never the working copy; just deploy here refuses uncommitted UI assets for that reason.

Secrets live in the age-encrypted secrets store (~/projects/data): sovrn/shared/<name> and sovrn/hosts/<fqdn>/<name>. Colmena decrypts them on the deploying machine and uploads them to /var/lib/sovrn-keys; nothing secret enters the Nix store or a repository. Values sovrn can generate come from apps.gen-secret (called by the fleet's gen-host-secrets); provider keys (object store, Resend, SMTP2GO) are stored by hand.

arm64 cells. Cells may be Hetzner CAX (aarch64). The deploying machine builds aarch64 under qemu emulation or hands builds to the staging cell mx99 as a Nix remote builder (/etc/nix/machines); production cells reuse the same store paths, so they never build anything. Only sovrn's Go commands are built; everything else comes from cache.nixos.org, and the vlpds image is fetched by digest on the deploying machine and shipped by Colmena.

Upgrades. nixpkgs is pinned in flake.lock and shared with moods and the fleet: just update-nixpkgs here and in moods, then just sync-nixpkgs in the fleet. Stalwart's version is asserted in nix/overlay.nix, so a nixpkgs bump that moves it fails evaluation until the pin is changed on purpose.

Verification #

  • nix build .#checks.x86_64-linux.<name>: VM tests for every role and module: cell, stalwart, stalwart-plan, metrics, relay (queue and drain), telemetry, backup, restore.
  • Staging first: deploy to mx99, test live (a test domain, mail in and out, the relay drain), then the production cells.
  • Recovery is rehearsed on mx99 (fleet plan, Phase 10).