# sov: design

sov turns one typed specification (an *instance* repository) into a running server and keeps it
that way. This file is the design; the code in this repository is the implementation.

## Goal

From "a VPS was just ordered" to "everything works" with one command, every time, with no step
that only exists in someone's memory.

- **Configuration** (OS, services, sites) comes from the instance repository.
- **State** (git repositories, statistics, server-only secrets) comes from the source bucket and
  is restored automatically on first boot.
- **Secrets** that are typed in or shared with outside services live encrypted in the instance
  repository; the owner's key unlocks them.

## Three places

| Place | Holds |
|---|---|
| The instance repository | the specification (`spec/`), the sites, the shared secrets, the machine (`host/`) |
| sov (this repository) | how everything is derived from a specification |
| The server's disk | the running system and the state |
| The source bucket (Hetzner Storage Box) | the system of record for the state: encrypted restic snapshots every 5 minutes, box snapshots on top |

Nothing else holds anything that matters. Certificates, DNS zones, healthchecks.io checks and the
GitHub mirror are derived: the server recreates them from the spec.

## Layers

| Layer | Tool | Lives in |
|---|---|---|
| Server OS, services, firewall, users | NixOS 26.05 (flake, pinned by `flake.lock`) | `modules/`, instance `host/` |
| Disk layout / first install | disko / nixos-anywhere | instance `host/disk.nix`, `tools/install.sh` |
| Updates | comin: pulls `main` (knot, GitHub mirror as fallback), accepts only commits signed by the owner's key | `modules/comin.nix` |
| HTTPS | Caddy, automatic certificates (Let's Encrypt, ZeroSSL fallback) | `modules/caddy.nix`, `modules/sites.nix` |
| DNS | Knot as hidden primary; Hetzner's three nameservers as secondaries (signed transfers and NOTIFY); CAA limits certificate issuers | `modules/dns.nix` |
| Git hosting | Tangled knot; private repos mirrored to GitHub | `modules/knot.nix`, `modules/mirror.nix` |
| Statistics | GoatCounter, fed from Caddy's access logs (no script on the pages) | `modules/stats.nix` |
| Backups | restic → Storage Box (5 min), daily prune + partial read check + test restore; restore on fresh install | `modules/backup.nix` |
| Server-only secrets | created once on the server, kept as state | `modules/secrets.nix` |
| Shared secrets | sops-nix (owner's SSH key and the server's host key as recipients) | instance `secrets/`, `.sops.yaml` |
| Alarms | healthchecks.io (synced from `spec.heartbeats`), email + ntfy push | `modules/heartbeats.nix` |
| Recovery drill | monthly: the server installs a copy of itself on a temporary Hetzner VM (nixos-anywhere + host key + restore from the bucket), compares it with itself, deletes the VM | `modules/drill.nix`, `modules/drill-target.nix` |
| Live checks | `invariants`, hourly: sites, certificates, redirects, DNS serials and delegation, knot, stats, mirror, deployed commit, disk | `modules/checks.nix` |

## The instance repository

```
spec/default.nix       the specification, checked against spec/schema.nix
sites/<name>/          each website's source with a package.nix (may use sov.staticSite, sov.fonts)
secrets/server.yaml    secrets the server reads (sops)
secrets/admin.yaml     secrets only the owner's tools read
secrets/host_key.sops  the server's SSH host key, its permanent identity; host_key.pub
host/default.nix       the machine: hardware, disk, network
.sops.yaml             who can decrypt what
flake.nix              outputs = { self, sov }: sov.lib.instance { inherit self; };
```

Every module declares the folders it keeps (`sov.state`); together they are what the source bucket
holds and what a fresh install restores. Users owning such folders have fixed ids, because restic
stores owners as numbers.

## Domains

Registrar: INWX, nameservers from `spec.dns.nameservers`. The parent zones' delegation is checked
by `invariants`; changing it is one of the steps in [operations.md](operations.md).

## Verification

- `nix flake check`: evaluates the spec, builds everything, boots the base config in a VM test.
- On the server: `invariants` (hourly, and on demand with `systemctl start invariants`).
- Recovery: daily, a restore into a scratch directory with database and repository checks; monthly,
  a full recovery drill onto a temporary VM (`systemctl start recovery-drill` runs one now).