sov: design #

sov turns one typed specification (an instance repository) into a running server and keeps it that way. This file is the design; the code in this repository is the implementation.

Goal #

From "a VPS was just ordered" to "everything works" with one command, every time, with no step that only exists in someone's memory.

• Configuration (OS, services, sites) comes from the instance repository.
• State (git repositories, statistics, server-only secrets) comes from the source bucket and is restored automatically on first boot.
• Secrets that are typed in or shared with outside services live encrypted in the instance repository; the owner's key unlocks them.

Three places #

Place	Holds
The instance repository	the specification (spec/), the sites, the shared secrets, the machine (host/)
sov (this repository)	how everything is derived from a specification
The server's disk	the running system and the state
The source bucket (Hetzner Storage Box)	the system of record for the state: encrypted restic snapshots every 5 minutes, box snapshots on top

Nothing else holds anything that matters. Certificates, DNS zones, healthchecks.io checks and the GitHub mirror are derived: the server recreates them from the spec.

Layers #

Layer	Tool	Lives in
Server OS, services, firewall, users	NixOS 26.05 (flake, pinned by flake.lock)	modules/, instance host/
Disk layout / first install	disko / nixos-anywhere	instance host/disk.nix, tools/install.sh
Updates	comin: pulls main (knot, GitHub mirror as fallback), accepts only commits signed by the owner's key	modules/comin.nix
HTTPS	Caddy, automatic certificates (Let's Encrypt, ZeroSSL fallback)	modules/caddy.nix, modules/sites.nix
DNS	Knot as hidden primary; Hetzner's three nameservers as secondaries (signed transfers and NOTIFY); CAA limits certificate issuers	modules/dns.nix
Git hosting	Tangled knot; private repos mirrored to GitHub	modules/knot.nix, modules/mirror.nix
Statistics	GoatCounter, fed from Caddy's access logs (no script on the pages)	modules/stats.nix
Backups	restic → Storage Box (5 min), daily prune + partial read check + test restore; restore on fresh install	modules/backup.nix
Server-only secrets	created once on the server, kept as state	modules/secrets.nix
Shared secrets	sops-nix (owner's SSH key and the server's host key as recipients)	instance secrets/, .sops.yaml
Alarms	healthchecks.io (synced from spec.heartbeats), email + ntfy push	modules/heartbeats.nix
Recovery drill	monthly: the server installs a copy of itself on a temporary Hetzner VM (nixos-anywhere + host key + restore from the bucket), compares it with itself, deletes the VM	modules/drill.nix, modules/drill-target.nix
Live checks	invariants, hourly: sites, certificates, redirects, DNS serials and delegation, knot, stats, mirror, deployed commit, disk	modules/checks.nix

The instance repository #

spec/default.nix       the specification, checked against spec/schema.nix
sites/<name>/          each website's source with a package.nix (may use sov.staticSite, sov.fonts)
secrets/server.yaml    secrets the server reads (sops)
secrets/admin.yaml     secrets only the owner's tools read
secrets/host_key.sops  the server's SSH host key, its permanent identity; host_key.pub
host/default.nix       the machine: hardware, disk, network
.sops.yaml             who can decrypt what
flake.nix              outputs = { self, sov }: sov.lib.instance { inherit self; };

Every module declares the folders it keeps (sov.state); together they are what the source bucket holds and what a fresh install restores. Users owning such folders have fixed ids, because restic stores owners as numbers.

Domains #

Registrar: INWX, nameservers from spec.dns.nameservers. The parent zones' delegation is checked by invariants; changing it is one of the steps in operations.md.

Verification #

• nix flake check: evaluates the spec, builds everything, boots the base config in a VM test.
• On the server: invariants (hourly, and on demand with systemctl start invariants).
• Recovery: daily, a restore into a scratch directory with database and repository checks; monthly, a full recovery drill onto a temporary VM (systemctl start recovery-drill runs one now).