Operations #

Everything done from outside the server is a command, run in the instance repository: nix run .#sov -- <command> (or sov <command> in its dev shell). sov help lists them. Each one is idempotent where it can be, never shows a secret, and makes its repository changes as signed commits that the server deploys by itself.

Day to day #

Change anything	edit the spec or a site, signed commit, git push
Is everything fine?	sov status (the live checks; they also run hourly and alarm on failure)
Does recovery work?	sov drill (also runs monthly and alarms on failure)

Secrets #

sov secrets lists every secret: what it is, where it lives, how a new one is made. They are declared next to the code that uses them (sov.secrets in the modules).

sov rotate <secret>... replaces secrets; sov rotate --all replaces every secret kept in git: the response to "a key leaked". Generated secrets need nothing from you. For secrets of outside services (healthchecks.io, Hetzner, Bluesky) it opens the right page, tells you what to click, and takes the new value in a hidden prompt. The order is always: the new value works before the old one stops.

Secret kind	What rotation does
restic password	the bucket accepts the new password, the server switches, the old one is removed
DNS secondaries secret	the server switches, then the secondaries (Hetzner API)
host key	both keys may decrypt and open the bucket, the server switches, the old key is removed
mirror deploy key	the server makes a new key; you register it on the mirror
pasted tokens	new token stored and deployed; you delete the old one on the service

The owner's key #

sov key rotate makes a new owner key (in the file git signs with), registers it everywhere (server login, deploy signatures, secrets, source bucket, Tangled), switches, and removes the old key everywhere. The one step left to you is printed: the key on GitHub.

If the owner's key is lost: use the paper key (below) to decrypt, reinstall from the source bucket (sov install, through the provider's rescue system), then sov key rotate and sov paper print.

The paper recovery key #

sov paper print makes a new recovery key (age): every secret is encrypted to it, and it opens the source bucket. The previous paper key stops working. It writes a printable sheet: the key in short lines with check codes and a QR code, plus the recovery steps for this instance. Print it, store it offline, delete the file.

sov paper check (yearly; the alarm reminds you): type the key in from the paper, line by line. It verifies the key decrypts the secrets and opens the bucket, and resets the reminder.

Once per new thing #

A repository on the knot	sov add repo <name> "<description>", then add the printed entry to the spec
A DNS zone	add it to the spec's dns.records, deploy, sov add zone <zone>, then point the domain's nameservers to spec.dns.nameservers
A domain moved to INWX	sov add domain --prices <domain>, --check, then without flags (asks for the INWX login and auth codes)
A GitHub mirror	repos.<name>.mirror = { owner; name; }, deploy, create the repository, register the key: ssh admin@<server> sudo ssh-keygen -y -f /var/lib/sov/secrets/mirror_key_<name> as deploy key with write access

Recovery #

sov install <user>@<ip> wipes the machine and installs the instance; on first boot it restores the state from the source bucket and starts with the same identity. The monthly drill does exactly this on a temporary VM. The paper sheet lists the steps for the case where everything else is lost.

The source bucket's door (in the box's .ssh/authorized_keys, kept by sov rotate host_key):

command="rclone serve restic --stdio /home/<dir>",restrict ssh-ed25519 <secrets/host_key.pub>