Hosted Service Runbook #
Operating procedures for the hosted Civility service: deploy, verify, roll back, back up, and respond to incidents. Written for the hosted production stack, but the procedures apply to any serious deployment — self-hosters can substitute their own platform for Deno Deploy.
Companion documents: admin panel (day-to-day operations), observability (OTel wiring), deployments (self-host templates).
1. Production topology #
civility.cloud app.civility.cloud
┌──────────────────┐ ┌──────────────────────┐
│ Deno Deploy │ │ Deno Deploy │
│ www/serve.ts │ │ CivServer (main.ts) │
│ static marketing │─────▶│ /api/v1, /admin, │
│ site, no state │ form │ /dashboard │
└──────────────────┘ post └───┬──────────┬───────┘
│ │
┌────────▼───┐ ┌───▼──────────┐
│ PostgreSQL │ │ S3-compatible│
│ (managed) │ │ blob storage │
└────────────┘ └──────────────┘
Two independent Deno Deploy projects:
| Project | Source | State |
|---|---|---|
civility.cloud |
www/serve.ts |
None — fully static; talks to the API only from the browser. |
app.civility.cloud |
CivServer entry | PostgreSQL (DATABASE_URL) + S3 (S3_*) + secrets below. |
Secrets live in Deno Deploy project settings (never in the repo): CIVILITY_AUTH_SECRET, CIVILITY_ADMIN_TOKEN, DATABASE_URL, S3_*, plus OTEL_* when telemetry is on. Email and billing (Stripe) credentials are not env vars — they are entered in /admin/email and /admin/billing and stored sealed in the database under CIVILITY_AUTH_SECRET. That makes CIVILITY_AUTH_SECRET the one unrecoverable secret: losing it invalidates sessions/JWTs and orphans every sealed credential and the services KEX key. Keep an offline copy.
2. Deploying #
Deploys ride Deno Deploy's git integration: a push to main builds and progressively replaces isolates. There is no separate build step for either project.
Pre-deploy checklist
deno task testgreen locally (Postgres suites too:PG_TEST_URL=... deno task test).- If the change adds a migration: confirm it is additive (see §3) and registered in both
adapters/db/sqlite/db.tsandadapters/db/postgres/db.ts. - If the change touches signup/billing/webhooks, note the affected flow — that is what you verify after deploy.
Post-deploy verification (2 minutes, every deploy)
curl -s https://app.civility.cloud/api/v1/doc | head -c 200— server up, version bumped.- Log in to
/dashboardwith a test account — auth path works. - Open
/admin/— panel up; glance at/admin/logsfor anything unusual. - If the deploy touched sync: push/pull once from a known app.
Rollback. Deno Deploy keeps previous deployments addressable: promote the prior deployment from the project dashboard (Deployments → … → Promote to production). Because migrations are forward-only (§3), the previous code version must tolerate the current schema — which additive migrations guarantee. If a bad migration itself is the problem, do not roll back the deploy; fix forward (write a corrective migration) and treat it as an incident.
3. Migrations #
Migrations run automatically at server boot (runMigrations inside createManager), tracked in the migrations table. Operational rules:
- Forward-only, additive. New tables, new nullable columns, new indexes. Never drop or rename in the same release that stops using a thing — old isolates run against the new schema during rollout, and rollback re-runs old code against it.
- Both dialects, always. Every migration exists as an SQLite and a PostgreSQL file with the same id; CI's SQLite suite plus a
PG_TEST_URLrun covers both. - Code backfills (like
010_auth_hardening's token hashing) must be idempotent — they re-run if the boot that ran them crashed before serving.
Disposable development databases during the encryption removal #
This pre-release breaking change edits the initial SQLite and PostgreSQL schemas instead of adding a compatibility migration. A database that already recorded the initial migration will not rerun it when its source changes. After the revised initial schemas land, start with a fresh development SQLite path (--db-path) or a new empty PostgreSQL database (DATABASE_URL / PG_TEST_URL) and let startup apply migrations there. Keep the previous database and blob directory until any development data you need has been saved; source changes do not delete either one. This recreation guidance is for disposable development environments, not an in-place production upgrade.
4. Backups & restore #
| Data | Mechanism | Drill |
|---|---|---|
| PostgreSQL | Provider's automated daily snapshots + PITR if available | Quarterly: restore latest snapshot to a scratch database, run civ api start against it, log in. |
| Blob objects (S3) | Provider versioning / lifecycle rules | Included in the DB drill: verify a known blob downloads through the restored server. |
| Secrets | Offline copy of CIVILITY_AUTH_SECRET (+ admin token) |
Confirm the offline copy matches on every rotation. |
A database restore without the matching CIVILITY_AUTH_SECRET is not a restore — sealed email/billing credentials and all sessions die with the secret. Verify both together.
5. On-call & support #
Rotation: one person (Ben). Writing it down is what makes it a rotation of one rather than an assumption.
- Response expectation: best effort, no SLA. Beta users are told this at invite time.
- Support channel: email (the direct beta feedback channel). Support requests that reveal defects go into the bug backlog; operational ones may become runbook sections.
- Daily (while beta runs): glance at
/admin/logs(auth failures,quota_exceededspikes) and the provider dashboards (Deno Deploy errors, Postgres storage/connections). - Away/unreachable: the service runs unattended; nothing needs a human on a timer. Crons (prune, blob GC, metering) are self-contained; their runs land in
/admin/logs.
6. Incident response #
An incident is anything that degraded the service for a user or put data at risk — including near-misses and self-inflicted deploy breakage. Small ones count; the point of the log is pattern-spotting.
First moves, in order:
- Scope it. One user or everyone?
/admin/logs+ OTel traces tell you which. - Check the platforms before debugging code: Deno Deploy status, Postgres provider status, S3 provider status. Most infrastructure incidents are theirs, not yours.
- Recent deploy? If the incident started at a deploy boundary, promote the previous deployment (§2 Rollback) first, diagnose second.
- Data at risk? If writes are corrupting (not just failing), take the service down deliberately — a hard outage beats silent corruption. Pause by promoting a known-good deployment or, worst case, suspending the Deno Deploy project.
Severity vocabulary (used in the log):
| Sev | Meaning |
|---|---|
| S1 | Data loss/corruption, or full outage |
| S2 | A core flow (signup, login, sync, billing) broken for all |
| S3 | Degraded: partial outage, one user blocked, slow |
| S4 | Near-miss / no user impact (caught in verification) |
7. Routine operations quick reference #
| Task | How |
|---|---|
| Mint a beta invite | /admin/invites → note + expiry → share the /signup?invite= link |
| Open/close the beta gate | /admin/invites → "Require an invite to sign up" |
| Raise one user's quota | /admin/users/:id → limit overrides |
| Change email transport | /admin/email (test-send button confirms) |
| Blob GC on demand | civ api gc blobs --once (or the admin handler) |
| Audit trail / support triage | /admin/logs — filter by user, action, date; CSV/JSON export |
| Verify a deploy | §2 post-deploy checklist |