nix/ #

Packages, NixOS modules, VM tests and dev runners for sovrn. (They replaced the Ansible deployment, removed in c3b0d04.)

Hosts are deployed from the fleet in ~/projects/servers (one Colmena hive that can run several projects on a machine; plan: docs/fleet-migration-plan.md there). sovrn exports what the fleet composes: nixosModules.services (options, secrets, the Stalwart/sovrnd/vlpds/backup/telemetry modules, the sovrn user), nixosModules.cell, .metrics, .relay, .pds, .telemetryEdge, and apps.gen-secret (per-host secret values). The host recipes in the Justfile are stubs that forward to the fleet; just fleet shows the layout. Deployment as a whole: docs/deployment.md.

Path	What
overlay.nix	pkgs.sovrn.*: Stalwart (nixpkgs stalwart_0_16, version-asserted), stalwart-cli, the vlpds image. Used by both flake.nix (hosts) and devenv.nix (dev).
bulwark-image.nix	The Bulwark webmail image every cell and dev run, pinned by digest per architecture. Bumping it is an edit here, then the webmail QA pass.
bulwark/	Bulwark's admin config, built here and mounted read-only (admin-config.nix): the settings policy, the installed plugins, the sovrn theme (theme/sovrn.css, accents only, forced on) and the logos and icons it serves at /api/admin/branding/, so nothing is clicked together in Bulwark's admin UI. environment.nix holds the branding settings (names, logo and icon URLs, the AGPL source link), shared by the cells and dev. plugins/sovrn-mail-apps is the "Mail apps" Settings page and plugins/sovrn-atmosphere the "Atmosphere Settings" one (for now the private copy, a link to the mailbox page in sovrn) (both hand-written CommonJS, no build step; per-cell values are written into the bundle). Adding a plugin: a directory with manifest.json and its entry file, listed in admin-config.nix.
vlpds-image.nix	The vlpds image every PDS node and dev run, upstream plus patches/vlpds, pinned per architecture. just vlpds-image-pin bumps it (and copy internal/pds/reserved.txt).
systemd/	The build machine's weekly patched vlpds image release, a systemd user timer (just vlpds-timer-install; patches/vlpds/README.md). Not part of any host.
pkgs/	Our own derivations: vlpds-image.nix (the pinned image as a docker archive, added to the deployer's store by just vlpds-image-pin), and go-commands.nix (sovrnd, backup-route-sync, metrics-healthz from one build). Built on the deployer; no binary cache.
fleet.nix	Defaults for sovrn.fleet, the settings every host shares: mail domain, root cell, ACME, relay, PDS. A fleet host overrides them with settings.sovrn in the fleet's hosts.json.
modules/options.nix	sovrn.fleet, sovrn.cell.hostname, sovrn.metrics.hostname: what the modules read instead of module arguments, so sovrn can share a host with other projects.
modules/services.nix	sovrn's own pieces on any host: options, secrets, the Stalwart/sovrnd/vlpds modules (off until a role enables them), the sovrn user. Exported as nixosModules.services.
modules/stalwart.nix	sovrn.services.stalwart: pinned Stalwart, declared config.json, recovery admin and plan secrets as systemd credentials, sandboxed unit. Every start (boot, deploy, restart) runs Stalwart in recovery mode, applies the plan with stalwart-converge, then starts normal mode. The unit counts as started once normal mode is ready. Enabled on cells.
modules/sovrnd.nix	sovrn.services.sovrnd: the control plane, with sovrn.toml generated from settings (file paths only, no secrets). Starts after Stalwart is converged and ready.
modules/pds.nix	sovrn.pds: a node of the region's vlpds cluster (the image under podman), its WireGuard link to the region's other nodes, and its Caddy sites (the PDS name and every handle host). See "The PDS" below.
modules/webmail.nix	sovrn.webmail: the cell's Bulwark (ADR-0010), the pinned image under podman on loopback, served by Caddy at webmail.<cell> with its admin and setup surfaces refused (except the reads every user's client makes: policy, plugin bundles, theme CSS, branding). Its Account Security tab is off (STALWART_FEATURES=false); app passwords and mail app setup are on sovrn's mailbox page. Signs in through the cell's sovrnd (OIDC); talks JMAP to the cell's Stalwart through the cell host (cell-edge.nix proxies /jmap, with CORS for the webmail). Its synced user settings are backed up with the cell.
modules/mailfront.nix	sovrn.mailfront: the shared sign-in at mail.<zone> on the landing host, and customers' mail.<domain> CNAMEs (on-demand certificates; it answers Caddy's ask and passes handle hosts on to vlpds). Its domain → cell cache is rebuilt from the cells, so it isn't backed up.
modules/secrets.nix	sovrn.secrets.<name>: secrets a role needs, uploaded by Colmena from the secrets store. Colmena key attributes and units are prefixed sovrn- (files are not), so other projects' keys on the same host can't collide; order services after .unit.
modules/backup.nix	sovrn.backup: Litestream and rclone to the cell's bucket in its region's object store (sovrn.fleet.objectStores.<region>), freshness markers for /healthz, the backup guard, automatic restore (sovrn-restore) and sovrn-freshen-backups. Enabled on cells.
modules/telemetry-edge.nix	sovrn.telemetry.edge: node_exporter, vmagent and Vector pushing to the metrics box, with the cell/host/service labels the alert rules use.
modules/roles/	cell runs Stalwart, sovrnd, the Caddy edge (cell-edge.nix), backups and the telemetry edge; metrics and relay (the backup MX, see below). scratch is a placeholder.
tests/	NixOS VM tests, exposed as flake checks (cell, stalwart, metrics, relay, telemetry, backup, restore; need KVM; relay takes about 6 minutes because it waits for Stalwart's real first retry), plus the fast stalwart-plan check: nix build .#checks.x86_64-linux.<name> -L. tests/node.nix is the test nodes' base.
stalwart/	Stalwart's configuration as stalwart-cli apply plans: readable JSON arrays, composed per deployment in stalwart/default.nix (see below).
scripts/	Dev process runners started by devenv up; gen-secret.sh.

nixpkgs and the Stalwart pin #

Hosts and dev use the same nixpkgs revision. flake.lock pins it, and devenv.yaml repeats the revision. Bump both with just update-nixpkgs (then the same in moods, and just sync-nixpkgs in the fleet).

nixos-unstable is the only channel that ships Stalwart 0.16 (26.05 has 0.15.5). A bump that moves Stalwart off the version in overlay.nix fails evaluation until the pin is changed on purpose. Stalwart comes from cache.nixos.org for x86_64 and aarch64, so nothing is built from source.

Stalwart plans #

Stalwart's server configuration is checked in as apply plans under stalwart/. Each file is a JSON array of operations with a _comment explaining each block. default.nix composes them per deployment, in apply order:

File	Contents
common.plan.json	Stdout tracer, Prometheus metrics, the admin web UI disabled (sovrnd fronts everything), DKIM signing off (only the outbound relay signs).
cell.plan.json	Let's Encrypt provider, the cell's service domain (sovrn.internal, carrying the MX certificate), server identity, the cell listener set.
dev.plan.json	Dev's domain, identity and unprivileged listeners.
sendlimits.plan.json	Every inbound throttle, including the beta send limits: 100 remote recipients per UTC day per account and 300 per sender domain (bug 93dd413). Cells and dev only.
events.plan.json	The signed event webhook to sovrnd's loopback /internal/stalwart/events: senders and rate-limit hits, for the send-limit alerts (bug 93dd413). Cells and dev only.
roles.plan.json	Stalwart's standard roles and default role assignments. A declaratively created database has none, and without them new accounts have no permissions.
sovrnd.plan.json	sovrnd's service account and its restricted role.

sovrnd's role extends User and Tenant Administrator (Stalwart won't let an account grant permissions it lacks) and adds:

• the registry objects sovrnd manages, plus impersonate and authenticate
• not System Administrator: no system Sieve scripts, hooks/milters/webhooks, stores, listeners, certificates or tracing

A new feature that needs more is a reviewed one-line change here.

nix/scripts/stalwart-render-plan.sh turns plans into what apply reads:

• @name@ tokens are host values (hostname, service domain, ACME).
• {"$secret": "name"} placeholders are filled from key files at converge time, so secrets never enter the store.
• It refuses anything left unresolved.

The Stalwart module's stalwart-converge [URL] renders a host's plan and applies it as the recovery admin. Every operation is an upsert, reconcile or update, so re-running it is safe.

stalwart.service runs it on every start:

1. Recovery mode on :18080 (no MTA, management API only).
2. stalwart-converge.
3. Stop recovery mode, then start normal mode, which binds the listeners the plan just defined.
4. Wait for /healthz/ready.

Changing a plan changes the converge script, so a deploy restarts Stalwart and applies it. Mail is down for the restart (senders retry, and the backup relay queues). If the converge fails, the unit retries every 30s indefinitely. Check journalctl -u stalwart, where the converge output appears.

nix build .#checks.x86_64-linux.stalwart-plan -L applies every composition twice to a real Stalwart inside the build sandbox, with pebble standing in for Let's Encrypt. It fails on unknown properties, bad references or a second apply that isn't a no-op. It takes seconds, so run it after editing a plan.

Dev Stalwart #

devenv up runs Stalwart natively (scripts/serve-stalwart.sh), bootstrapping the same way hosts will:

1. Write a declared config.json (sqlite store only).
2. Start in recovery mode on :18080.
3. stalwart-cli apply of the dev composition from stalwart/default.nix, the same files and renderer the hosts use.
4. Restart in normal mode on :8080.

The apply only re-runs when the plan (or the binary) changes. just dev runs sovrnd as the plan's sovrnd@sovrn.internal service account, like the hosts. A missing permission therefore shows up in dev first as a forbidden error; add it to sovrnd.plan.json. To start over: sovrn-reset.

Hosts #

Hosts are installed and deployed from the fleet (~/projects/servers); see its README. Roles come from the fleet's hosts.json (sovrn-cell, sovrn-metrics, sovrn-relay), not from the hostname. A cell is exclusive; metrics (and the relay) share the netcup box with moods.

cd ~/projects/servers
just new-host <ipv4> <fqdn> <hetzner|netcup> <roles>   # ERASES the box
just deploy-project sovrn                             # sovrn's last commit to every host running it

The fleet deploys sovrn's last commit (jj: @-), never the working copy: commit first. just deploy here does that, after checking that the embedded UI assets are committed.

Colmena connects by hostname, so a host needs DNS and an SSH config entry that picks the key. One wildcard block covers sovrn's hosts:

Host *.sovrn.at
    User root
    IdentityFile ~/.ssh/hetzner
    IdentitiesOnly yes

Secrets #

Secrets live in the age-encrypted secrets store (SECRETS_DIR, default ~/projects/data), one file per value, in the exact form the service reads:

Store path	Contents
sovrn/shared/<name>	Fleet-wide values: OIDC and OAuth keys, backup-domains token, relay, Resend, metrics. The relay API key, Resend key and metrics password are symlinks to the provider entries already in the store.
sovrn/hosts/<fqdn>/<name>	Per-host values: SSH host key, Stalwart recovery admin, sovrnd's Stalwart account, postmaster, object-store key (s3-access-key/s3-secret-key, the cell's bucket); on PDS nodes the WireGuard key (wg-pds.key) and the vlpds peer certificate (vlpds-peer.{crt,key}, from just pds-peer-tls).
sovrn/hosts/pds.<region>.sovrn.at/<name>	A region's vlpds cluster, the same on every node (on the host as vlpds-<region>-<name>): jwt-secret, admin-token, internal-token, kek, plc-rotation-key (nix run .#gen-secret -- kek pds.eu.sovrn.at makes each), s3-access-key/s3-secret-key (the region's PDS bucket only), email-smtp-url, and the peer CA (peer-ca.crt deployed, peer-ca.key never).

A role declares what it receives in modules/roles/*.nix, e.g. sovrn.secrets."oidc-key.pem" = { scope = "shared"; user = "sovrn"; };. A service reads config.sovrn.secrets.<name>.path.

On deploy, Colmena runs secrets decrypt on the deploying machine and uploads the result to /var/lib/sovrn-keys/<name> (0400). That directory is persistent, not /run/keys, so secrets survive a reboot. Nothing secret enters the Nix store or this repo.

Every key uploads before activation, so a service the switch restarts finds its key, and a new key needs no separate colmena upload-keys. On a fresh host the owner (e.g. sovrn) doesn't exist yet at upload: Colmena leaves the file root-owned and chowns it during activation, once users exist. The owner must therefore be a declared NixOS user (secrets.nix asserts it); DynamicUser services take keys root-owned via LoadCredential. (Post-activation upload used to hang the first deploy declaring a service-owned key: the switch waited on the service, the service on the key.)

Recipe	What it does
gen-host-secrets <fqdn> (fleet)	Creates the host's missing per-host values; sovrn's come from nix run .#gen-secret -- <name> <fqdn>. new-host runs it. Provider values it can't generate (object-store keys) are listed with the secrets encrypt command to store them.
check-secrets [fqdn] (fleet)	Fails if any declared secret is missing. deploy runs it first.

CI #

None for now: tests and deploys run from the operator's machine. Staging (mx99.eu.sovrn.at) gets a change first, production cells after it looks right.

The backup relay #

modules/roles/relay.nix (sovrn.relay.*) is the pri-20 MX. It can share a machine; the fleet runs it on the netcup box next to moods.

Piece	How it runs
Stalwart	relay.plan.json: :25 (the MX) and loopback http only, its own hostname as its only domain (certificate by ACME through Caddy's port-80 challenge proxy). Port 25 checks SPF/DMARC against the real sender, as an MX does.
backup-route-sync	Polls each cell's GET /backup/domains (shared bearer token) every 60 s and owns the per-cell Relay routes, the delivery schedule (TTL queueDays, backoff 5m → 72h), the outbound strategy and the RCPT relaying exception. It logs in as its own restricted account (route-sync@<relay hostname>, role in the plan), not the recovery admin. Without cells it doesn't run, and Stalwart relays for nobody.

Cells take relayed mail on 2525, not 25. Stalwart treats every port but 25 as submission, and on 25 it checks SPF, DMARC and reverse IP against the connecting IP, which for relayed mail is the relay's. Delivered on 25, legitimate mail queued during an outage would be recorded as SPF/DMARC failures (relaxed mode doesn't reject, but the spam filter sees them and DMARC failure reports go out). The cell's relay listener (sovrn.cell.relayPort) is open only to sovrn.cell.relaySources, needs no auth, skips those checks (the relay already ran them) and still adds a Received header and runs the spam filter. The relay therefore needs no outbound port 25.

The registry client (internal/stalwart) sends no accountId: Stalwart resolves it to the caller. The fixed "a" it used to send only worked for callers holding impersonate.

The cell #

modules/roles/cell.nix puts these on a cell (plus backups and the telemetry edge):

Piece	How it runs
Stalwart	Converges its plan on every start (see above). Gets its own certificate by ACME HTTP-01, answered through Caddy's port-80 challenge proxy.
sovrnd	sovrnd.service as the sovrn user, reading /etc/sovrn/sovrn.toml. Talks to Stalwart as the restricted sovrnd@sovrn.internal account.
Caddy	The cell's hostname serves sovrnd. On the root cell, sovrn.at also serves the landing page and did.json.

With the fleet's sovrn-pds role beside it, sovrnd uses the region's vlpds node on the same box (pds.url on the WireGuard link): onboarding a domain adds at.<domain> as a vlpds handle domain at runtime and never changes the Nix config. A domain's *.at handle CNAME points at the region's PDS name (pds.eu.sovrn.at), so <user>.at.<domain> reaches vlpds's /.well-known/atproto-did.

The PDS #

modules/pds.nix (sovrn.pds, the fleet's sovrn-pds role) runs one node of a region's vlpds cluster. A region has two: its cell and its backup host, on the region's PDS bucket in the region's object store. There's no quorum (leases and shard ownership are compare-and-swap writes in the bucket). It runs vlpds's tiny profile with two shards, one per node: vlpds creates an account only in a shard the receiving node owns, so each node needs one.

Piece	How it runs
vlpds	podman-vlpds.service: the pinned image, loaded from the store, on the host network bound to the node's WireGuard address only. Secrets are mounted read-only from the key directory. A stop is graceful (90 s): the node hands its shard to the peer.
WireGuard	wg-pds, the region's link (10.77.0.0/24, UDP 51820): node-to-node traffic (vlpds mTLS on top), Caddy's peer upstream and the ACME hop.
Caddy	pds.<region>.sovrn.at and every other HTTPS name (handle hosts, on-demand certificates that vlpds's /tls-check approves) go to the local node first and the peer's second, so a node that's down costs no client error while its box is up. Both boxes are in DNS and each holds its own certificates; an HTTP-01 challenge this Caddy didn't start is forwarded to the peer's, once. updateHandle (403) and the operator paths (404) are refused.

The operator console is on the node's link address: ssh -L 2583:10.77.0.1:2583 root@<host>, then http://localhost:2583/admin.

After a failover both shards stay on the node that took them until the other node is back and joined; then the fair share (one each) hands one back.