Secrets management with sops-nix #

Mental model #

Secrets live encrypted in this repo and are decrypted on the host at activation time, landing as root-owned files under /run/secrets/. Nothing secret ever enters the nix store or git history.

No dedicated age keys exist. Every identity is an existing SSH ed25519 key, re-expressed in age's format with ssh-to-age (a deterministic conversion, not a new key):

Keypair	Created by	Private half lives	sops role
Personal key	me, `ssh-keygen`	my workstations	admin: edit/recover any secret
Host key (per host)	sshd, first boot	`/etc/ssh/ssh_host_ed25519_key`	decrypt that host's secrets at activation
Signing key (per host)	minted at onboarding	that host's sops secrets	payload: signs commits made on the host
Forge auth key (per host)	minted at onboarding	that host's sops secrets	payload: pushes to git forges (Tangled, Forgejo, ...)

Signing and forge-auth keys are deliberately separate: auth keys are disposable (revoke on a forge anytime, nothing else breaks), signing keys are historical (forges verify old commits against currently-registered keys, so they stay registered even after a host is gone). One key per host per role keeps revocation surgical.

.sops.yaml (repo root) maps recipients to paths: everything under secrets/<host>/ is encrypted to the admin key and that host's key. Declared secrets (sops.secrets.<name> in hosts/<host>/default.nix) are decrypted by sops-nix during activation — no extra key provisioning, the host key is already there.

Knot 2 keeps its master key in a separate file, secrets/brume/knot.yaml (the knot_env key, holding KNOT_MASTER_KEY). It is declared with an explicit sopsFile and must be backed up off-server: the knot's sealed repository-signing keys cannot be recovered without it.

Agent forwarding does not work for decryption. Signing asks the agent to perform an operation; age decryption needs the raw private key bytes, which agents never expose. Run sops where a private key actually lives.

Editing secrets #

On a host (uses the host's own key, hence sudo; the key must first be converted to an age identity — sops does not read host keys directly):

sudo env SOPS_AGE_KEY="$(sudo ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key)" \
  sops secrets/brume/secrets.yaml

(sops and ssh-to-age are in the devShell; outside it, prefix with nix shell nixpkgs#sops nixpkgs#ssh-to-age -c.)

That is bash syntax — in nushell, "$(...)" is a literal string and sops will report "failed to load age identities". From nu, wrap the whole thing in bash instead:

sudo nix shell nixpkgs#sops nixpkgs#ssh-to-age -c bash -c 'SOPS_AGE_KEY="$(ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key)" sops secrets/brume/secrets.yaml'

(root's $EDITOR is usually unset, so sops falls back to vi; prepend EDITOR=nano inside the quotes for nano.)

On a workstation holding the personal key: no env vars needed — sops automatically tries ~/.ssh/id_ed25519 as an identity:

sops secrets/brume/secrets.yaml

The file opens as plain YAML in $EDITOR; it re-encrypts on save.

Adding a new secret #

Add the key/value in the secrets file (see above).

Declare it in the host config:

sops.secrets.my-secret = { };                      # root:root, mode 0400
sops.secrets.my-secret = { owner = "puregarlic"; } # if a user/service reads it

Reference it by path, never by value — secrets are runtime files, not eval-time strings:
```
services.something.keyFile = config.sops.secrets.my-secret.path;
```
git add any new file (flakes ignore untracked files — evaluation will fail with a "path does not exist" style error until tracked).
nixos-rebuild test, verify, switch.

Onboarding a new host #

Two-pass flow. A brand-new host has no secrets/<host>/ directory and declares no secrets, so its first install is an ordinary deploy with nothing to decrypt — the "partial" state is just a host that hasn't been given secrets yet.

Install and boot the host from this config (add hosts/<name>/, hardware config, etc.). sshd generates its host key on first boot.

Harvest the host's age recipient:

ssh-keyscan <host> | ssh-to-age
# or on the host itself:
ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub

Register it in .sops.yaml: add the key under keys: with an anchor, and a creation_rules entry for secrets/<host>/.* listing [*admin_puregarlic, *host_<name>].
Create and populate secrets/<host>/secrets.yaml (encryption only needs public recipients, so this works from anywhere):
```
sops secrets/<host>/secrets.yaml
```
Declare sops.defaultSopsFile + secrets in hosts/<name>/default.nix, git add, rebuild the host.
Mint the host's identity keys (signing + forge auth): generate each with ssh-keygen -t ed25519, store private halves in the host's secrets file (signing_key, forge_auth_key), declare them with owner = "puregarlic". The signing public key also goes in modules/keys/signing.nix; the forge-auth public key only goes to the forges (upload as auth/deploy key on Tangled, Forgejo, etc. — and the signing key as a verification key where the forge supports it).

One-pass alternative for later: pre-generate the host keypair on the admin machine, register its recipient before install, and inject the private key to /etc/ssh/ssh_host_ed25519_key during installation (what nixos-anywhere's --extra-files is for). sshd keeps a pre-existing key instead of generating one, so the machine can decrypt from its very first boot.

Adding a letta agent #

Agents are declared as drop-in files under modules/nixos/letta/agents/ (one file per agent; auto-imported). Each agent likewise gets its own sops file, secrets/brume/agents/<agent>.yaml, holding up to three keys — signing_key + forge_auth_key (skip both, and step 1's keygen, for forgeIdentity = false agents), appserver_token (appServer.enable), and env (secretEnv: KEY=value lines of agent-private environment variables, loaded by the agent's services and login shells). The module maps agent files to runtime paths /run/secrets/<agent>_<key>.

Model providers are registered with letta once, on a template agent, then cloned: letta connect writes richer state than plain API key env vars unlock, so that resulting file is the artifact we share. It lives in the host's main secrets.yaml as letta_provider_state (group letta, like other shared agent secrets), and garden.letta.providerStatePath tells the module where it belongs inside each agent's home.

Mint the agent's keys (workstation or host):

ssh-keygen -t ed25519 -N "" -C "brume-<agent>-signing" -f ./<agent>_signing
ssh-keygen -t ed25519 -N "" -C "brume-<agent>-forge"   -f ./<agent>_forge
head -c 32 /dev/urandom | base64   # app-server capability token

Store the private halves via sops secrets/brume/agents/<agent>.yaml (see "Editing secrets"; the secrets/brume/.* creation rule covers new files here) as signing_key and forge_auth_key (multiline | blocks), and the token as appserver_token. Shred the local key files afterward.
Register the public halves: signing pubkey goes in modules/keys/signing.nix as brume-<agent>; forge-auth pubkey is uploaded to the forges only (same split as host keys — signing keys are historical, auth keys disposable). If the agent commits as its own DID (did in the agent file), also register the DID in modules/keys/allowed-signers.nix under agentDids.brume-<agent> — the agent file references that map, and it is what makes the agent's signatures verify (and attribute correctly) everywhere.
Declare the agent: copy an existing file in modules/nixos/letta/agents/, pick a fresh uid (1100+) and app-server port, git add both the agent file and its secrets file, nixos-rebuild test → switch. Secrets are declared automatically from the agent entry.
Bootstrap once as the agent (sudo -iu <agent>):
```
letta-seed                           # install shared provider state
letta channels configure <channel>   # per channel, interactive
letta server --backend local --channels <list> --install-channel-runtimes
```
The last command pulls channel runtimes into ~/.letta; Ctrl-C once it is up. Verify letta does not demand a Letta Cloud login in local mode — if it does, whatever credential lands in ~/.letta needs a decision (leave imperative, or promote to a sops secret).

On the template agent (or after adding a provider), the provider-state flow is instead: letta backend local, then letta connect <provider> ... per provider, then harvest ~/.letta/lc-local-backend/providers/auth.json into sops — paste it into secrets.yaml as letta_provider_state and point garden.letta.providerStatePath (modules/nixos/letta/shared.nix) at its home-relative path. Other agents then pick it up via letta-seed (-f to overwrite).

Local-backend provider notes: the provider list differs from the api backend (letta connect with no args shows it). There is no openai-compatible here, but together and fireworks are first-party, and openai, llama-cpp, and lmstudio all accept --base-url + --api-key for arbitrary OpenAI-compatible endpoints. One registration per provider slot — a second connect openai overwrites the first (--name is not honored as a separate registry key), so spread custom endpoints across the openai / llama-cpp / lmstudio slots.
Set autoStart = true; in the agent's file and rebuild (or systemctl start letta-server-<agent> letta-app-server-<agent>).

Reinstalled host #

A reinstall mints a fresh host key: SSH clients warn REMOTE HOST IDENTIFICATION HAS CHANGED, and the old recipient in .sops.yaml now points at a private key that no longer exists. Nothing is lost — the admin key still decrypts everything. Re-onboard like a new host:

Harvest the new recipient (step 2 above).
Replace the host's old age key in .sops.yaml.
Re-encrypt existing files for the new recipient set:
```
sops updatekeys secrets/<host>/secrets.yaml
```
Rebuild the host.

updatekeys is also the step after any .sops.yaml recipient change — editing the policy file alone does not re-encrypt anything.

Recovery properties #

The personal key can decrypt every secret: losing a host loses no data.
A host key only decrypts that host's own secrets: a compromised host leaks its secrets, not the fleet's.
Secrets stored in sops (e.g. per-host signing keys) survive reinstalls; anything derived from the host key itself (its age identity) does not.