Lith recovery #

DigitalOcean droplet ac-lith (561482364, sfo3) uses daily disk backups, with seven days of retention and an 08:00–12:00 UTC start window. At the current $24/month droplet price, daily backups add $7.20/month. An on-demand snapshot is also copied to nyc3 before maintenance. Snapshots cost $0.06/GB/month; adding a region does not increase that charge. See backup pricing and snapshot pricing.

lith-backup-prepare.timer runs at 07:30 UTC. It exports PostgreSQL roles and every connectable non-template database into root-only /var/backups/lith/. Custom-format exports are checked with pg_restore --list and checksummed before publication. Completed exports older than seven full days are pruned only after a successful run. The disk backup then includes these consistent exports alongside the rest of the server. Failed exports fail the systemd unit; monitor its status and the DigitalOcean backup inventory.

Install as root from /opt/ac:

install -m 755 lith/backup-prepare.sh /usr/local/sbin/lith-backup-prepare
install -m 644 lith/lith-backup-prepare.{service,timer} /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now lith-backup-prepare.timer
systemctl start lith-backup-prepare.service
systemctl status lith-backup-prepare.service lith-backup-prepare.timer

For recovery, create a replacement droplet from the chosen snapshot or daily backup. Keep it isolated from public traffic and outbound mail initially: restored credentials and timers are live, including repository mirroring. Stop Lith, mail, feed, IPFS, Oxigraph and mirror/sync timers until the clone is ready. Check Caddy routing, SSH access, environment files, external MongoDB and Spaces connectivity, and local databases before switching DNS.

If PostgreSQL needs logical recovery, select a completed export directory, verify sha256sum -c SHA256SUMS, and restore roles from globals.sql only into a fresh cluster. Create the target database and restore its .dump with pg_restore --exit-on-error. Test into a uniquely named empty database first; never use --clean against production as a validation step. The globals file includes sensitive role information: keep all exports private.

These backups cover Lith's local disk. External MongoDB, Spaces, other droplets, and attached volumes need their own backups. This is recovery capacity, not automatic failover. Daily backups remain in the source region; the additional-region snapshot is a manual checkpoint and does not refresh with each daily backup. Refresh it before substantial maintenance and remove superseded snapshots only after verifying the replacement.