Connected apps with 3rd-party data hosts jsr.io/@civility
civility docs hosting runbook.md
9.8 kB · 135 lines
Markdown
at main

Hosted Service Runbook #

Operating procedures for the hosted Civility service: deploy, verify, roll back, back up, and respond to incidents. Written for the hosted production stack, but the procedures apply to any serious deployment — self-hosters can substitute their own platform for Deno Deploy.

Companion documents: admin panel (day-to-day operations), observability (OTel wiring), deployments (self-host templates).


1. Production topology #

civility.cloud            app.civility.cloud
┌──────────────────┐      ┌──────────────────────┐
│ Deno Deploy      │      │ Deno Deploy          │
│ www/serve.ts     │      │ CivServer (main.ts)  │
│ static marketing │─────▶│ /api/v1, /admin,     │
│ site, no state   │ form │ /dashboard           │
└──────────────────┘ post └───┬──────────┬───────┘
                              │          │
                     ┌────────▼───┐  ┌───▼──────────┐
                     │ PostgreSQL │  │ S3-compatible│
                     │ (managed)  │  │ blob storage │
                     └────────────┘  └──────────────┘

Two independent Deno Deploy projects:

Project Source State
civility.cloud www/serve.ts None — fully static; talks to the API only from the browser.
app.civility.cloud CivServer entry PostgreSQL (DATABASE_URL) + S3 (S3_*) + secrets below.

Secrets live in Deno Deploy project settings (never in the repo): CIVILITY_AUTH_SECRET, CIVILITY_ADMIN_TOKEN, DATABASE_URL, S3_*, plus OTEL_* when telemetry is on. Email and billing (Stripe) credentials are not env vars — they are entered in /admin/email and /admin/billing and stored sealed in the database under CIVILITY_AUTH_SECRET. That makes CIVILITY_AUTH_SECRET the one unrecoverable secret: losing it invalidates sessions/JWTs and orphans every sealed credential and the services KEX key. Keep an offline copy.


2. Deploying #

Deploys ride Deno Deploy's git integration: a push to main builds and progressively replaces isolates. There is no separate build step for either project.

Pre-deploy checklist

  1. deno task test green locally (Postgres suites too: PG_TEST_URL=... deno task test).
  2. If the change adds a migration: confirm it is additive (see §3) and registered in both adapters/db/sqlite/db.ts and adapters/db/postgres/db.ts.
  3. If the change touches signup/billing/webhooks, note the affected flow — that is what you verify after deploy.

Post-deploy verification (2 minutes, every deploy)

  1. curl -s https://app.civility.cloud/api/v1/doc | head -c 200 — server up, version bumped.
  2. Log in to /dashboard with a test account — auth path works.
  3. Open /admin/ — panel up; glance at /admin/logs for anything unusual.
  4. If the deploy touched sync: push/pull once from a known app.

Rollback. Deno Deploy keeps previous deployments addressable: promote the prior deployment from the project dashboard (Deployments → … → Promote to production). Because migrations are forward-only (§3), the previous code version must tolerate the current schema — which additive migrations guarantee. If a bad migration itself is the problem, do not roll back the deploy; fix forward (write a corrective migration) and treat it as an incident.


3. Migrations #

Migrations run automatically at server boot (runMigrations inside createManager), tracked in the migrations table. Operational rules:

  • Forward-only, additive. New tables, new nullable columns, new indexes. Never drop or rename in the same release that stops using a thing — old isolates run against the new schema during rollout, and rollback re-runs old code against it.
  • Both dialects, always. Every migration exists as an SQLite and a PostgreSQL file with the same id; CI's SQLite suite plus a PG_TEST_URL run covers both.
  • Code backfills (like 010_auth_hardening's token hashing) must be idempotent — they re-run if the boot that ran them crashed before serving.

Disposable development databases during the encryption removal #

This pre-release breaking change edits the initial SQLite and PostgreSQL schemas instead of adding a compatibility migration. A database that already recorded the initial migration will not rerun it when its source changes. After the revised initial schemas land, start with a fresh development SQLite path (--db-path) or a new empty PostgreSQL database (DATABASE_URL / PG_TEST_URL) and let startup apply migrations there. Keep the previous database and blob directory until any development data you need has been saved; source changes do not delete either one. This recreation guidance is for disposable development environments, not an in-place production upgrade.


4. Backups & restore #

Data Mechanism Drill
PostgreSQL Provider's automated daily snapshots + PITR if available Quarterly: restore latest snapshot to a scratch database, run civ api start against it, log in.
Blob objects (S3) Provider versioning / lifecycle rules Included in the DB drill: verify a known blob downloads through the restored server.
Secrets Offline copy of CIVILITY_AUTH_SECRET (+ admin token) Confirm the offline copy matches on every rotation.

A database restore without the matching CIVILITY_AUTH_SECRET is not a restore — sealed email/billing credentials and all sessions die with the secret. Verify both together.


5. On-call & support #

Rotation: one person (Ben). Writing it down is what makes it a rotation of one rather than an assumption.

  • Response expectation: best effort, no SLA. Beta users are told this at invite time.
  • Support channel: email (the direct beta feedback channel). Support requests that reveal defects go into the bug backlog; operational ones may become runbook sections.
  • Daily (while beta runs): glance at /admin/logs (auth failures, quota_exceeded spikes) and the provider dashboards (Deno Deploy errors, Postgres storage/connections).
  • Away/unreachable: the service runs unattended; nothing needs a human on a timer. Crons (prune, blob GC, metering) are self-contained; their runs land in /admin/logs.

6. Incident response #

An incident is anything that degraded the service for a user or put data at risk — including near-misses and self-inflicted deploy breakage. Small ones count; the point of the log is pattern-spotting.

First moves, in order:

  1. Scope it. One user or everyone? /admin/logs + OTel traces tell you which.
  2. Check the platforms before debugging code: Deno Deploy status, Postgres provider status, S3 provider status. Most infrastructure incidents are theirs, not yours.
  3. Recent deploy? If the incident started at a deploy boundary, promote the previous deployment (§2 Rollback) first, diagnose second.
  4. Data at risk? If writes are corrupting (not just failing), take the service down deliberately — a hard outage beats silent corruption. Pause by promoting a known-good deployment or, worst case, suspending the Deno Deploy project.

Severity vocabulary (used in the log):

Sev Meaning
S1 Data loss/corruption, or full outage
S2 A core flow (signup, login, sync, billing) broken for all
S3 Degraded: partial outage, one user blocked, slow
S4 Near-miss / no user impact (caught in verification)

7. Routine operations quick reference #

Task How
Mint a beta invite /admin/invites → note + expiry → share the /signup?invite= link
Open/close the beta gate /admin/invites → "Require an invite to sign up"
Raise one user's quota /admin/users/:id → limit overrides
Change email transport /admin/email (test-send button confirms)
Blob GC on demand civ api gc blobs --once (or the admin handler)
Audit trail / support triage /admin/logs — filter by user, action, date; CSV/JSON export
Verify a deploy §2 post-deploy checklist

See also #