monitor #

Health-endpoint monitor on Cloudflare Workers + Durable Objects. See DESIGN.md for the brief.

• One MonitorDO per region (wnam, enam, weur, eeur, apac), placed with location hints. Each runs an alarm every CHECK_INTERVAL_SECONDS, GETs every endpoint, and stores results in SQLite (a trigger drops rows older than 14 days).
• Every check is reported to one global AlertsDO, which alone sends alerts, deduped across regions: one once a region has seen FAILURE_THRESHOLD consecutive failures (a single blip, e.g. a short deploy, never alerts), one every REMINDER_HOURS while any region is still failing, and one on recovery, once every failing region has seen 2 consecutive 200s. A region that stops reporting for 3 check intervals no longer holds an incident open. Anything other than HTTP 200 (including timeouts) is a failure. Alerts go to ALERT_TO.
• Each alert also goes to Pushover once its secrets are set: a second, independent channel. Each channel tracks its own last alert, so a failed send is retried on that channel only. Pushover messages list the failing entries of a /healthz checks object; recoveries arrive at priority -1 (quiet).
• An hourly cron (and any homepage visit) makes sure every region's alarm loop is running.
• https://monitor.rtw.run/ shows active alarms and the latest result per endpoint; /endpoint?url=… shows history.
• An alarm can be silenced from the homepage: no more reminders until it recovers (the recovery is still sent). Silencing covers what is failing at that moment (failing /healthz check names, else the HTTP status or "no response"); anything else failing ends the silence and alerts immediately.
• If an endpoint is removed from ENDPOINTS, its open alarm can't recover, so the homepage offers Dismiss instead: that deletes the incident and its per-region state. Re-adding the endpoint later starts a fresh incident (two consecutive failures, as usual). Configured endpoints cannot be dismissed.

Configuration #

Endpoints, interval, timeout and alert settings are vars in wrangler.jsonc; change them and redeploy. Basic-auth credentials are secrets (all requests are rejected until both are set):

npx wrangler secret put AUTH_USER
npx wrangler secret put AUTH_PASS

Pushover is off until both of its secrets are set (an application token and your user key):

npx wrangler secret put PUSHOVER_TOKEN
npx wrangler secret put PUSHOVER_USER

Development #

pnpm test        # vitest in the Workers runtime
pnpm typecheck
pnpm dev         # local server; credentials come from .dev.vars (AUTH_USER=…, AUTH_PASS=…)
pnpm deploy
pnpm types       # regenerate worker-configuration.d.ts after editing wrangler.jsonc