Deploying grain-serve #
grain-serve and its pages run from one image, the root's Dockerfile, as two
services of the compose project in deploy/docker-compose.yml: serve,
grain-serve on 2600, and pages, the SvelteKit app in web/ under
adapter-node on 2601. deploy/deploy.sh builds the image on the host and
rolls both over with deploy/rollout.sh, as knotless's deploy does.
cp deploy/local.env.example deploy/local.env # GRAIN_HOST, GRAIN_DOMAIN, ...
just deploy # deploy/deploy.sh
just logs serve 'select(.level == "WARN")' # deploy/logs.sh
main deploys itself: on each push, .knotless/workflows/deploy.yml runs
deploy/deploy.sh build on the host while the commit's checks run, then
deploy/deploy.sh up once check, test, web and lexicons have all
passed, or deploy/deploy.sh drop when one fails or main has moved on.
Its env: is production's deploy/local.env: deploy.sh writes the host's
/etc/grain-serve/.env from it, whole, so a setting left out there is
unset on the host.
On grain's host #
grain's host (~/code/hetzner-infra, stacks/grain) runs one compose
project in /etc/grain: Caddy, the PDS, imgproxy and porxie. grain-serve
is a project of its own beside it, in /etc/grain-serve, as knotless's is
in /etc/knotless: deploy.sh writes /etc/grain-serve/.env and copies
docker-compose.yml and rollout.sh there, and keeps its state in
GRAIN_DATA (/mnt/data/grain-serve): index.db, and the push key files
under certs/.
The secrets are on the host alone, in /etc/grain-serve/serve.env
(mode 600), never in the repository or the deploy's environment:
GRAIN_CDN_KEY=<hex> # imgproxy's IMGPROXY_KEY
GRAIN_CDN_SALT=<hex> # imgproxy's IMGPROXY_SALT
GRAIN_APNS_KEY_FILE=/data/certs/AuthKey.p8
GRAIN_FCM_KEY_FILE=/data/certs/firebase.json
CLOUDFLARE_ACCOUNT_ID=<id>
CLOUDFLARE_AI_TOKEN=<token> # Workers AI only, for Clef
What is not a secret is a flag in docker-compose.yml, set from .env
(GRAIN_JETSTREAM, GRAIN_BACKFILL, GRAIN_CDN, GRAIN_APNS_*, ...:
deploy.sh's header lists them). Empty turns a part off.
The proxy #
TLS is Caddy's, in hetzner-infra. It reaches grain-serve on the external
docker network GRAIN_PROXY_NETWORK (grain-serve), which deploy.sh
makes, by the alias grain-serve. hetzner-infra's compose puts Caddy on
that network, as it puts it on knotless, and its Caddyfile sends grain's
domain there:
grain.social, www.grain.social {
import cf_tls
reverse_proxy grain-serve:2600
encode gzip zstd
}
grain-serve hands the pages, on the project's own network, every request
it does not answer itself (--web http://pages:2601), as knotless-serve
does, so the proxy knows of nothing else.
Both services are reached by name, which docker's DNS answers with every running container's address, so a rollout's two containers both answer until the old one stops, and a dial to one not listening goes to the other.
Deploys without downtime #
rollout.sh pages serve moves each service in turn: a second container
starts beside the running one, and the old one is stopped once the new
one's healthcheck (/healthz for serve, /_app/version.json for pages)
passes, or the new one is removed and the old one kept if it never does
within GRAIN_ROLLOUT_TIMEOUT (180s). serve is rolled only if the new
build keeps the index as it is, which grain-serve --check-index /data/index.db says: a build whose migrations would change the tables
under the old one stops the old one first, down until the new one
listens; a build older than the index is not started.
Monitoring #
/healthzanswersokwhile grain-serve listens: what docker's healthcheck and rollout.sh wait on./_statusanswers 200, or 503, with JSON saying what does not hold: Jetstream connected and bringing events in the last two minutes, the index answering a read, and the disk under it with room (under 15% free is told of, under 5% fails). It says theversionandcommitrunning, the commit being the one deploy.sh built (GRAIN_COMMIT). It is never cached (no-store). A monitor off the host polls it.
Logs #
serve and pages log to the host's journal (the journald driver,
x-logging in docker-compose.yml), so a container's lines outlive it.
Each names its project and service (COM_DOCKER_COMPOSE_PROJECT=grain-serve,
COM_DOCKER_COMPOSE_SERVICE). deploy.sh and rollout.sh add a line with
the time for each step of a deploy to /var/log/grain-serve-deploy.log.
What serve logs, and /metrics and PUT /log on --metrics
(127.0.0.1:9464, inside the container), is in
docs/observability.md.
just logs serve --since 2h 'select(.level == "WARN" or .level == "ERROR")'
just logs pages --follow
just logs deploys