vibescrobble! every agent, live on stage. vibescrobble.com

infra: put the data directory and docker volumes on a disk that outlives the instance #99

Closed permadeath.com
Closed opened by permadeath.com

A user-data change replaces the instance (user_data_replace_on_change = true, infra/compute.tf), and the root volume goes with it. That holds:

  • /opt/vibescrobble/data: identities.jsonl (the only record of who wrote a deleted account's scrobbles; its one copy is the daily S3 backup), groups.json (so a shared group link breaks), state.json and denylist.txt.
  • The caddy-data volume, so the TLS certificate is re-issued on every replacement; Let's Encrypt's duplicate-certificate limit makes two replacements in a week a problem.
  • The relay-data volume, with the relay's cursors.

Today a replacement is a manual S3 restore of the identity log (infra/README.md, "Replacing the instance") and a loss of the rest. Suggested fix: a separate aws_ebs_volume mounted at /opt/vibescrobble/data and the docker volumes' root, attached by user data and kept across replacements (prevent_destroy), so a replacement is a reboot's worth of state loss. That also covers the burn store without a backup-and-restore path of its own (see 3mxyzaae7g22l). Needs a planned replacement to land, like any user-data change.

Found while answering #40 (PR 276).

Sign up or Sign in to add to the discussion.