# Operations

Everything done from outside the server is a command, run in the instance repository:
`nix run .#sov -- <command>` (or `sov <command>` in its dev shell). `sov help` lists them. Each one is
idempotent where it can be, never shows a secret, and makes its repository changes as signed
commits that the server deploys by itself.

## Day to day

| | |
|---|---|
| Change anything | edit the spec or a site, signed commit, `git push` |
| Is everything fine? | `sov status` (the live checks; they also run hourly and alarm on failure) |
| Does recovery work? | `sov drill` (also runs monthly and alarms on failure) |

## Secrets

`sov secrets` lists every secret: what it is, where it lives, how a new one is made. They are
declared next to the code that uses them (`sov.secrets` in the modules).

`sov rotate <secret>...` replaces secrets; `sov rotate --all` replaces every secret kept in git: the
response to "a key leaked". Generated secrets need nothing from you. For secrets of outside services
(healthchecks.io, Hetzner, Bluesky) it opens the right page, tells you what to click, and takes the
new value in a hidden prompt. The order is always: the new value works before the old one stops.

| Secret kind | What rotation does |
|---|---|
| restic password | the bucket accepts the new password, the server switches, the old one is removed |
| DNS secondaries secret | the server switches, then the secondaries (Hetzner API) |
| host key | both keys may decrypt and open the bucket, the server switches, the old key is removed |
| mirror deploy key | the server makes a new key; you register it on the mirror |
| pasted tokens | new token stored and deployed; you delete the old one on the service |

## The owner's key

`sov key rotate` makes a new owner key (in the file git signs with), registers it everywhere (server
login, deploy signatures, secrets, source bucket, Tangled), switches, and removes the old key
everywhere. The one step left to you is printed: the key on GitHub.

If the owner's key is lost: use the paper key (below) to decrypt, reinstall from the source bucket
(`sov install`, through the provider's rescue system), then `sov key rotate` and `sov paper print`.

## The paper recovery key

`sov paper print` makes a new recovery key (age): every secret is encrypted to it, and it opens the
source bucket. The previous paper key stops working. It writes a printable sheet: the key in short
lines with check codes and a QR code, plus the recovery steps for this instance. Print it, store
it offline, delete the file.

`sov paper check` (yearly; the alarm reminds you): type the key in from the paper, line by line.
It verifies the key decrypts the secrets and opens the bucket, and resets the reminder.

## Once per new thing

| | |
|---|---|
| A repository on the knot | `sov add repo <name> "<description>"`, then add the printed entry to the spec |
| A DNS zone | add it to the spec's `dns.records`, deploy, `sov add zone <zone>`, then point the domain's nameservers to `spec.dns.nameservers` |
| A domain moved to INWX | `sov add domain --prices <domain>`, `--check`, then without flags (asks for the INWX login and auth codes) |
| A GitHub mirror | `repos.<name>.mirror = { owner; name; }`, deploy, create the repository, register the key: `ssh admin@<server> sudo ssh-keygen -y -f /var/lib/sov/secrets/mirror_key_<name>` as deploy key with write access |

## Recovery

`sov install <user>@<ip>` wipes the machine and installs the instance; on first boot it restores
the state from the source bucket and starts with the same identity. The monthly drill does exactly
this on a temporary VM. The paper sheet lists the steps for the case where everything else is lost.

The source bucket's door (in the box's `.ssh/authorized_keys`, kept by `sov rotate host_key`):

```
command="rclone serve restic --stdio /home/<dir>",restrict ssh-ed25519 <secrets/host_key.pub>
```