# comail Operator Runbook

Runbook for operators of a comail relay. Covers routine
maintenance, common incidents, and the "known weirdness" that has bit us
during development so future sessions don't waste time re-discovering it.

Scope: the relay service itself, the Osprey worker, the labeler, the
relay's supporting DNS, and the three provider FBL integrations. Does
not cover atproto PDS operation (separate stack).

> **Configuration:** Examples below use placeholder host names that you
> should replace for your deployment:
>
> - `{RELAY_HOST}` — your relay's reachable hostname (e.g. `relay.example.com`)
> - `{RELAY_IP}` — your relay's outbound IPv4 (RFC 5737 `203.0.113.77` is used in examples)
> - `{OPS_HOST}` — your ops/observability host
> - `{KAFKA_HOST}` — host running the Osprey Kafka broker
> - `{LABELER_HOST}` — host running the atproto labeler
>
> `ssh root@{RELAY_HOST}` means SSH to your relay VPS. Substitute the
> literal value (or, better, set these as environment variables in your
> operator shell profile).

---

## Topology one-liner

```
member SMTP client (swaks, app)
        │  587/STARTTLS + SASL PLAIN
        ▼
{RELAY_HOST} (VPS, {RELAY_IP}, Tailscale)
  │  signs mail with d=<member-domain> (alignment) + d=atmos.email (pool FBL)
  │  stamps X-Atmos-Member-Did + Feedback-ID
  │  emits relay_attempt → Kafka
  ▼
fan-out to recipient MX
  │
  └─► Kafka ({KAFKA_HOST}) ──► Osprey worker (SML rules) ──► Kafka (execution_results)
                                                          │
                                                          ▼
                                         {RELAY_HOST} consumer goroutine
                                                          │
                                                          ▼
                                            relay.sqlite.relay_events
                                                          │
                                                          ▼
                                         /admin/events, /admin/members/{did}
```

FBL complaints (ARF) and bounces (DSN) arrive on smtp.atmos.email MX,
classified, written to `relay.sqlite.inbound_messages`, attributed to
the originating member via X-Atmos-Member-Did, and fired as
`complaint_received` / `bounce_received` Osprey events.

---

## Routine tasks

### 1. Rotate operator DKIM key (atmos.email)

The operator keypair signs every outbound message with d=atmos.email. A
rotation is warranted on suspected private-key compromise, the ~annual
cadence recommended by M3AAWG, or if you simply want to force receivers
to re-evaluate the key.

```bash
# 1. On the relay host, rotate the key file. The relay regenerates on next
#    startup if the file is missing.
ssh root@{RELAY_HOST} rm /var/lib/atmos-relay/operator-dkim-keys.json
ssh root@{RELAY_HOST} systemctl restart atmos-relay

# 2. Grep the journal for the new public halves.
ssh root@{RELAY_HOST} "journalctl -u atmos-relay --since '2 min ago' | grep operator_dkim.first_boot"

# 3. Publish the new records at atmos<YYYYMMDD>{r,e}._domainkey.atmos.email
#    in Bunny (edit infra/dns.tf, open PR).

# 4. Leave the OLD records in DNS for ~48 h. Any mail signed with the
#    prior key still in flight needs its public half to verify. After
#    48 h, drop the old records in a follow-up PR.
```

### 2. Rotate member DKIM key

Triggered by member re-enrollment or explicit rotation request.

```bash
# Member rotates the DKIM key from /account/domains/{domain} (the
# /enroll wizard was retired in Funnel cleanup #525, 2026-05-15). The
# relay generates a fresh RSA + Ed25519 pair, returns the public
# halves on the rotation result page, hashes the API key, and stores
# the private halves in relay.sqlite member_domains.dkim_{rsa,ed}_privkey
# columns.
#
# Operator publishes the two TXT records at
#   atmos<YYYYMMDD>r._domainkey.<member-domain>
#   atmos<YYYYMMDD>e._domainkey.<member-domain>
#
# Labeler (on {LABELER_HOST}) verifies DKIM-in-DNS before issuing
# verified-mail-operator. It retries on failure, so publish → ≤ 60 s
# → labeled.
```

### 3. Approve a pending member

Pending members exist but cannot send (rejection: `535 5.7.8 pending
operator approval`). Approval is manual — shared-IP reputation requires
operator scrutiny of every new sender.

```bash
# Via admin API (Tailscale-only)
ADMIN=<admin-token-from-sops>
curl -X POST -H "Authorization: Bearer $ADMIN" \
  http://{RELAY_HOST}:8080/admin/member/<did>/approve

# Or via the /admin/members dashboard, click Approve on the pending row.
```

Before approving: check `/admin/members/<did>` (the rich detail page).
Green on DKIM, attestation, labels, reasonable send-activity history →
approve. Anything suspicious (scraped-looking domain, no attestation
record, broken DKIM) → hold and investigate.

### 4. Emergency suspend a member

```bash
curl -X POST -H "Authorization: Bearer $ADMIN" \
  "http://{RELAY_HOST}:8080/admin/member/<did>/suspend?reason=<url-encoded>"
```

SMTP submission from that DID starts rejecting immediately with
`535 5.7.8 account suspended`. Osprey event `relay_rejected` with
`reject_reason=suspended` fires for each attempt, so the Osprey UI
surfaces suspension activity.

### 5. Tune warming tier caps

Defaults live in `internal/relay/warming.go` `DefaultWarmingConfig`:

| Tier | Age | Hourly | Daily |
|---|---|---|---|
| warming | 0..7 d | 5 | 20 |
| ramping | 7..14 d | 20 | 100 |
| warmed  | 14+ d | member's configured HourlyLimit/DailyLimit | " |

Tier caps compose with per-member limits via `min()` — a tighter
per-member configured cap always wins; tier caps never raise a
member's limit.

Two labels override the tier:
- `highly_trusted` (issued by Osprey's reputation rules) skips
  warming entirely.
- `burst_warming` halves the effective hourly cap during the current
  window even for warmed senders.

To change the defaults globally, edit `DefaultWarmingConfig()` and
redeploy. Per-member exceptions: stay with the existing
`HourlyLimit`/`DailyLimit` columns; those compose.

### 6. Route operator-classified mail to an external inbox

Mail addressed to operator aliases on `atmos.email` (`postmaster@`,
`abuse@`, `fbl@`, `support@`, `hostmaster@`) is normally accepted and
dropped at the relay — the inbound classifier logs it and moves on.
When `operatorForwardTo` is set in the relay config, those messages
are instead re-sent through the same reply-forwarder used for
per-member reply routing, landing in whatever inbox you configure.

This is the escape hatch for provider authorization flows: Microsoft
SNDS enrollment, Google Postmaster domain ownership emails, abuse
contact probes from CERTs, and anything else that expects a human to
read mail sent to `postmaster`.

```bash
# Current forward destination (NixOS-managed)
ssh root@{RELAY_HOST} "cat /var/lib/atmos-relay/config.json | jq -r .operatorForwardTo"

# Change it: edit infra/nixos/default.nix, commit, let CI deploy.
# The field is config.json `operatorForwardTo`, set from the NixOS
# module. Restart is not zero-downtime for in-flight SMTP but the
# queue drains on boot.
```

Wiring lives in `internal/relay/inbound.go` (`processOperator`
handler) and `cmd/relay/main.go` (`SetOperatorForwarding` call). When
the field is empty, the old accept-and-log behaviour is preserved.
Verified by `TestInbound_OperatorForward_DeliversPostmasterMail` and
`TestInbound_OperatorForward_DisabledPreservesAcceptAndDrop`.

Testing the forward end-to-end:

```bash
# Send a test to any operator alias.
swaks --server smtp.atmos.email --ehlo smtp.atmos.email \
      --from ops-probe@example.com --to postmaster@atmos.email \
      --header "Subject: operator-forward smoke test" --body "ping"

# Confirm the forward ran.
ssh root@{RELAY_HOST} "journalctl -u atmos-relay --since '2 min ago' \
  | grep -E 'inbound\.operator|reply_forwarder\.sent'"
```

The forwarded message preserves the original From: and Subject:
headers so the downstream inbox sees the provider's original sender
(e.g. `notifications@microsoft.com`), not the relay.

---

## FBL integrations

comail runs one pool-level FBL registration for each of the
three providers that offer one, anchored to the operator-DKIM domain
(`atmos.email`) and the Feedback-ID `mailer` field
(`comail`). Complaints flow to `fbl@atmospheremail.com`
(Porkbun forwarder → Proton alias `atmosphere.support@scottlanoue.com`).

Every outbound message carries:
- DKIM d=member-domain (DMARC alignment with From:)
- DKIM d=atmos.email (pool FBL routing)
- X-Atmos-Member-Did: <did> (attribution)
- Feedback-ID: transactional:<did>:<member-domain>:comail (Gmail)

### Current registration status

| Provider | Status | Evidence / next step |
|---|---|---|
| Gmail Postmaster Tools | Verified | TXT token published for `atmos.email`; dashboard live at postmaster.google.com. Reputation score needs ~48 h of sending volume to populate. |
| Microsoft SNDS | IP registered, authorization email landed via operator-forwarder | The enrollment flow required receiving a verification mail at `postmaster@atmos.email` — handled by the operator-forwarder routing described in section 6. |
| Microsoft JMRP | Registered | FBL recipient `fbl@atmospheremail.com` accepted. First complaint probe will confirm the delivery path. |
| Yahoo CFL | Verified 2026-04-20 | Domain verified via TXT (`yahoo-verification-key=…`) at the atmos.email apex. Verification record is a no-op now; tracked for removal in internal ref. Complaints will arrive at `fbl@atmospheremail.com` once Yahoo begins sending. |

Adding a new provider later: publish the FBL recipient as
`fbl@atmospheremail.com` if they accept an external address, otherwise
a per-provider mailbox. Any new verification mail to an operator
alias is handled by the forwarder from section 6 — no code change
required.

### Gmail Postmaster Tools
- URL: https://postmaster.google.com
- Domain: atmos.email
- Verification: TXT record at atmos.email apex (Google provides token)
- Once verified → domain reputation + spam rate metrics visible in
  the dashboard. FBL signals fold into the Feedback-ID attribution.

### Microsoft JMRP (Junk Mail Reporting Program)
- URL: https://sendersupport.olc.protection.outlook.com/pm/Postmaster
- Register sender: atmos.email
- FBL recipient: fbl@atmospheremail.com
- Microsoft mails a verification token within 24–48 h.

### Yahoo CFL (Complaint Feedback Loop)
- URL: https://senders.yahooinc.com/complaint-feedback-loop/
- Form submission: domain atmos.email, DKIM selectors
  atmos<YYYYMMDD>{r,e}, FBL recipient fbl@atmospheremail.com, contact
  postmaster@atmospheremail.com.
- Manual review, 1–5 day turnaround.

### Verifying complaints flow end to end

```bash
# 1. Confirm inbound MX accepts mail for atmos.email (Porkbun fwds
#    the atmospheremail.com aliases; atmos.email MX is smtp.atmos.email).
ssh root@{RELAY_HOST} "nix-shell -p swaks --run 'swaks --server smtp.atmos.email --to postmaster@atmos.email --helo smtp.atmos.email --from test@atmos.email --body hi 2>&1 | tail -5'"

# 2. Watch inbound_messages for the ARF row.
ssh root@{RELAY_HOST} "nix-shell -p sqlite --run 'sqlite3 /var/lib/atmos-relay/relay.sqlite \"SELECT id,classification,member_did,subject FROM inbound_messages ORDER BY id DESC LIMIT 5\"'"
```

---

## Known weirdnesses (discovered the hard way)

### Druid is gone, don't bring it back

The original Osprey stack included Druid for analytics. At our event
volume (<1000/day), Druid's publishing pipeline added 2–18 minutes of
query latency and 6 containers of operational surface with no upside.
Retired in homelab-nix PR #677. The relay now consumes
`osprey.execution_results` from Kafka directly into its own SQLite and
serves `/admin/events` from there. Don't resurrect Druid.

### Osprey worker Postgres sink is flaky

`OSPREY_EXECUTION_RESULT_STORAGE_BACKEND=postgres` does not reliably
write every event to Postgres. Events do reach Kafka reliably.
Don't trust `stored_execution_result` on the worker's Postgres as a
source of truth for anything user-visible; the relay's
`relay_events` table is authoritative.

### Porkbun email forwarding needs FQDN EHLO

A send to Porkbun's MX with bare-hostname EHLO (e.g.
`EHLO {RELAY_HOST}`) gets `504 5.5.2 need fully-qualified hostname` and
silently bounces. swaks defaults to `hostname(1)`, which is the relay's
short hostname on the VPS. Pass `--ehlo smtp.atmos.email` for any
script that sends to Porkbun forwards.

### Tangled stale branch cache

The Tangled mirror's frontend caches branch lists outside the knot's
git state. Deleting a branch at the git level (ls-remote confirms)
does not cause Tangled's UI to forget it. Escalate to Tangled
operators for a manual reindex if the dropdown bothers you.

### Gitea `run: |` blocks use dash

Bash-only array syntax (`ALIASES=(a b c)`) fails inside a workflow's
`run:` step because the runner shells out to `sh`, which is `dash` on
Debian slim. Use POSIX patterns (`for x in a b c`).

### Auth LOGIN vs PLAIN

The relay advertises only `AUTH PLAIN` after STARTTLS. swaks defaults
to LOGIN. Members pasting the quickstart verbatim get
`Auth not attempted, requested type not available`. The Step 5 swaks
block on the enroll success page now hard-codes `--auth PLAIN`.

### Em-dashes in Gitea workflow comments

One workflow we wrote broke with `import_bunny_record: command not
found` when a comment block between function use-sites contained an
em-dash (U+2014). Generator's tokenizer choked on it. Stick to ASCII
hyphens inside `run: |` blocks.

### Templ `+` in text nodes

A templ file containing `<strong>Week 3+:</strong>` fails to parse
— templ treats the `+` as an expression terminator. Rephrase
("Week 3 onward") or wrap in raw string delimiters.

---

## Incident response

### "Mail is not being delivered"

```bash
# Order of investigation:
# 1. Is the relay running?
ssh root@{RELAY_HOST} "systemctl status atmos-relay --no-pager | head -5"

# 2. Is SMTP reachable from the member's vantage?
swaks --server smtp.atmos.email --port 587 --tls --quit-after FIRST-HELO

# 3. Did the member's AUTH succeed? (check relay journal)
ssh root@{RELAY_HOST} "journalctl -u atmos-relay --since '15 min ago' | grep smtp.auth"

# 4. Did the message reach Kafka?
ssh root@{KAFKA_HOST} "docker exec osprey-kafka-XXX kafka-console-consumer --bootstrap-server localhost:9092 --topic osprey.actions_input --from-beginning --timeout-ms 5000 | grep <did>"

# 5. Is the downstream MX reachable?
ssh root@{RELAY_HOST} "timeout 10 nc -zv gmail-smtp-in.l.google.com 25"

# 6. Check delivery.result log line for the specific message_id.
```

### "Everything is in spam"

Expected on day 1 for a new member. See the Step 6 copy on the
enroll success page. Real mitigation:
1. Member engages recipients (open, not-spam, reply)
2. Confirm DKIM + SPF + DMARC all pass in a received message's headers
3. Check Google Postmaster Tools domain reputation; target is
   `Medium` or better by week 2
4. Check Microsoft SNDS for the relay IP's listing

### "Member is suspended but shouldn't be"

```bash
# Reactivate
curl -X POST -H "Authorization: Bearer $ADMIN" \
  http://{RELAY_HOST}:8080/admin/member/<did>/reactivate

# Investigate what rule suspended them
ssh root@{RELAY_HOST} "nix-shell -p sqlite --run 'sqlite3 /var/lib/atmos-relay/relay.sqlite \"SELECT verdicts, raw FROM relay_events WHERE sender_did = \\\"<did>\\\" AND action_name = \\\"member_suspended\\\" ORDER BY id DESC LIMIT 1\"'"
```

### "DKIM verification fails at the receiver"

```bash
# Confirm DNS has the record the relay signed with.
MEMBER_DOMAIN=scottlanoue.com
SELECTOR=atmos20260418
for v in r e; do
  echo "=== ${SELECTOR}${v}._domainkey.${MEMBER_DOMAIN} ==="
  dig +short TXT "${SELECTOR}${v}._domainkey.${MEMBER_DOMAIN}" @8.8.8.8
done

# Confirm the relay is using that selector.
ssh root@{RELAY_HOST} "nix-shell -p sqlite --run 'sqlite3 /var/lib/atmos-relay/relay.sqlite \"SELECT domain, dkim_selector FROM member_domains WHERE domain = \\\"${MEMBER_DOMAIN}\\\"\"'"

# If selector in DB does not match DNS, either publish the correct
# selector or trigger a member re-enrollment to sync them.
```

---

## Outside operator setup (first-time bootstrap)

`infra/.sops.yaml` in this repo ships with **placeholder age keys**
(strings starting with `age1example…`). They are syntactically valid
so the file parses, but no private key exists for them — they cannot
decrypt anything. The real recipients for our deployment live in the
sops metadata header of each `secrets/*.yaml` file, not in `.sops.yaml`.

If you are deploying your own comail instance, regenerate the keys
before the first `sops` invocation:

```bash
# 1. Admin/workstation key (kept on your dev host).
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt
# Save the printed "public key: age1..." line; that's what goes in .sops.yaml.

# 2. Host keys for relay + ops + staging.
#    sops-nix can derive an age key from each host's SSH ed25519 host
#    key automatically — easiest path is to provision the VMs first,
#    then on each host:
#      nix run nixpkgs#ssh-to-age -- < /etc/ssh/ssh_host_ed25519_key.pub
#    That prints the age public key. Copy it into the appropriate
#    key group in infra/.sops.yaml.
#
#    For atmos-staging specifically: that guest uses a STANDALONE
#    keypair (not derived from a host SSH key) so it can pre-exist
#    before the staging VM boots. Generate it with `age-keygen`, keep
#    the public half in .sops.yaml, and plumb the private half into
#    the guest via whatever host-side secret store you control (in
#    our deployment, homelab-nix common.yaml + virtiofs-share at boot).

# 3. Replace every `age1example…` string in infra/.sops.yaml with the
#    matching public key from steps 1-2. The four roles are admin,
#    relay host, ops host, staging guest.

# 4. Re-encrypt the secrets to the new recipients. You must already
#    hold a working private key for the secrets you start from — i.e.
#    use your own encrypted content, not the upstream repo's.
sops updatekeys infra/secrets/relay.yaml
sops updatekeys infra/secrets/ops.yaml
sops updatekeys infra/secrets/staging.yaml

# 5. Commit the updated .sops.yaml and re-encrypted secrets.
```

If you skip this step and try to `sops -d` a secret with only the
placeholder keys, decryption fails — the placeholders have no private
half. The first deploy will not be able to read the secrets and the
relay will fail to start.

---

## Deploy + rollback

The relay auto-deploys on merge to `main` via Gitea Actions (see
`.gitea/workflows/relay-deploy.yml`). Rollback path:

```bash
# Revert the offending commit and push
cd ~/repos/comail
git revert <sha>
git push gitea main

# Or deploy a specific prior tag
gh pr create --title "Rollback to <sha>" --body "..." && ...
```

Relay restart is zero-downtime for accepted-but-unflushed mail because
messages are persisted in the local queue dir before handoff; the
queue is drained by the new process on boot. In-flight SMTP
connections die and the sending client retries.

---

## Operator accounts

| Service | Login | Purpose |
|---|---|---|
| Hetzner | scott@... | Relay VPS hosting |
| Bunny DNS | scott@... | atmos.email + scottlanoue.com zones |
| Porkbun | scott@... | atmospheremail.com + threadline.tools |
| Gmail Postmaster | atmospheremail.ops@gmail.com | FBL for atmos.email |
| Microsoft JMRP | atmospheremail.ops@outlook.com | FBL for atmos.email |
| Yahoo CFL | form-based (no account) | FBL for atmos.email |
| Proton | atmosphere.support@scottlanoue.com | All operational inbound mail |

Password-manager all of these. Grant co-operators via password manager
share, not by resetting passwords.

---

## Admin token rotation

The relay admin API accepts a primary token via `ADMIN_TOKEN` and an
optional second-active token via `ADMIN_TOKEN_PREV`. Both are
constant-time compared on every request, so timing cannot leak which
slot accepted the call. The previous-token slot exists so an operator
can roll a new token without a flash-cutover of every downstream
consumer.

Rotation procedure (zero-downtime):

1. Mint the new token (e.g. `openssl rand -base64 48`).
2. Update sops/secrets with `ADMIN_TOKEN=<new>` AND
   `ADMIN_TOKEN_PREV=<old>`. Deploy.
3. Update every consumer of the admin token — CI secrets, dashboard
   credentials, the labeler bridge, scripts. Both values are accepted
   throughout this window.
4. Once every consumer reports the new token works, clear
   `ADMIN_TOKEN_PREV` from sops and redeploy. Old token stops working
   on that deploy.

Suspected token compromise short-circuits this: rotate immediately
without the dual-slot grace window — set only `ADMIN_TOKEN=<new>` and
deploy. The compromised token is invalid the moment the new relay
starts. A DB-backed multi-token primitive with revoke-by-id and
last-used tracking is the long-term answer; tracked separately as a
chainlink follow-up. The dual-slot design here is the safe stopgap.

## System-mail transport: Resend opt-in (internal ref)

User-facing system mail (signup verification, welcome, key-regen,
email-change verification, GDPR deletion notices) can route through
Resend's HTTP API instead of the relay's direct-MX path. This decouples
the signup funnel's deliverability from atmos.email's own warming
state — verification emails land in inbox via Resend's already-warmed
pool while atmos.email continues to warm on member-domain product
traffic.

**Operator-internal mail (operator-ping, FBL complaint relays)
ALWAYS uses direct-MX with d=atmos.email DKIM, regardless of this
flag.** That's the protocol/operator layer the comail rebrand
preserves untouched (internal ref).

### Pre-flight (one-time per Resend account)

1. Verify the sending domain in Resend's dashboard:
   - Add `comail.at` (or whichever domain matches `cfg.ResendFromAddress`)
   - Publish the DKIM TXT records Resend's dashboard generates (their
     selectors, not yours)
   - Publish SPF: `v=spf1 include:_spf.resend.com -all` on the apex
     (Resend's dashboard confirms the exact include)
   - Publish DMARC: start with
     `v=DMARC1; p=none; rua=mailto:postmaster@comail.at`
     to monitor for ~2 weeks, then upgrade to `p=quarantine`
2. Set up Cloudflare Email Routing (or equivalent) on comail.at so
   replies-to-system-mail land somewhere monitored:
   - Enable in Cloudflare dashboard for comail.at
   - Add wildcard route `*@comail.at` → your ops inbox
   - Click the verification email Cloudflare sends to the destination
3. Confirm `resend_api_key` is in `infra/secrets/relay.yaml`
   (sops-encrypted) — `sops infra/secrets/relay.yaml` to edit. The
   key surfaces as `RESEND_API_KEY` env on the relay via the existing
   `atmos-relay-env` template (see `infra/nixos/default.nix`).
4. Send a manual test from the Resend dashboard to confirm the
   verified domain works end-to-end before flipping the relay flag.

### Flip the transport (opt in)

Set `MAIL_TRANSPORT=resend` in the relay's environment (add to
`atmos-relay-env` template in `infra/nixos/default.nix`), then redeploy.
Optionally override `RESEND_FROM_ADDRESS` if you need a non-default
From; the default is `Comail <hello@comail.at>`.

Verify by watching journal:

```bash
ssh root@{RELAY_HOST} "journalctl -u atmos-relay --since '1 min ago' | grep opmail.resend"
```

Expected: `opmail.resend.enabled: from="Comail <hello@comail.at>" ...`
on startup. Then trigger a real signup (or `curl` a member suspend /
reactivate) and watch for `opmail.sent: transport=resend
message_id=r_<resend-id>` per dispatch. The Resend dashboard mirrors
that id and provides delivery / open / bounce telemetry per send.

### Roll back (revert to direct-MX)

Clear `MAIL_TRANSPORT` from the relay env (or set to `relay`), redeploy.
Subsequent system mail goes back through the d=atmos.email direct-MX
path. No data migration; the per-call routing decision happens each
send. Round-trip is ~3 min (build + nixos-rebuild + relay restart).

### Failure modes worth knowing

- **`opmail.resend.disabled: ... RESEND_API_KEY is empty`** — the env
  flag is on but no key in sops; the relay silently falls back to
  direct-MX. Set the key and redeploy.
- **4xx from Resend** (`Domain not verified`, `Invalid from address`) —
  Resend returns these when the sending domain hasn't passed their
  verification step. The error message lands in the relay log
  verbatim. Fix in the Resend dashboard, no redeploy needed.
- **5xx from Resend / timeout** — surfaces as a normal send error;
  the user-visible operation (signup, approve, regenerate) has
  already completed and the relay log captures the failure for
  follow-up. No auto-retry layer yet.
- **No per-transport metric label** — opmail metrics (`sent`,
  `failed`) are bumped the same way for both transports; the
  per-transport split lives in the log line (`transport=resend|relay`).
  Extend the `OpMailMetrics` interface if per-transport dashboards
  become a need.

## Encryption migrations (one-time)

Two idempotent admin endpoints rotate legacy plaintext columns into
AES-256-GCM blobs:

- `POST /admin/migrate-dkim-encryption`
- `POST /admin/migrate-forward-to-encryption`

Both are **default-off** — the handlers return 404 unless
`ENABLE_MIGRATION_ENDPOINTS=true` is set on the relay. This is
intentional: production scans probing `/admin/migrate-*` should see
nothing outside of the explicit migration window.

Procedure:

1. Confirm `RELAY_PII_KEY` is present in the relay's environment (the
   migrations require it to write the encrypted form).
2. Set `ENABLE_MIGRATION_ENDPOINTS=true` in the relay's environment and
   redeploy.
3. Run the migration:

   ```
   curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
     http://relay:8080/admin/migrate-dkim-encryption
   curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
     http://relay:8080/admin/migrate-forward-to-encryption
   ```

   Each returns `{"rotated": N, "skipped": M}`. The endpoints are
   idempotent — safe to re-run and converge to `rotated: 0`.
4. Clear `ENABLE_MIGRATION_ENDPOINTS` and redeploy. The endpoints
   return 404 again.

If the rotation reports an unexpected error (most commonly "no PIIKey
configured"), the underlying error is in the 500 response body and
also in relay stdout.

---

## What this runbook doesn't cover

- **atproto labeler operations**: separate stack ({LABELER_HOST}), its own
  state (`state_dir/labels.sqlite`). Labeler is mostly self-healing via
  jetstream retries; restart on crash and it catches up.
- **Kafka/Osprey worker maintenance**: documented in homelab-nix.
  Single node, KRaft mode, 30-day retention. Run
  `kafka-consumer-groups --describe` to spot stuck consumers.
- **Atmosphere Office**: entirely separate product. See
  `SPEC-atmosphere-office.md`.