Production service containers for rtw.run
README.md

RTW Services #

Production container images for the RTW app, built with Nix and pushed to atcr.io/rtw.run on the AT protocol container registry. App source lives in https://kilimanjaro.io/rtw`.

Images are built with dockerTools.buildLayeredImage and contain binaries + entrypoint scripts only. All heavy data (graphhopper graph-cache, geocoder parquet, tile archives) lives on volumes and is downloaded at container start when configured (from R2 for graphhopper, from upstream for waldo) — never baked into images.

Building and pushing #

Images are multiarch (amd64 + arm64; the production server is arm64). Both arches are built on the dev machine with Nix — no arm64 hardware needed.

nix build .#image-<service>            # host-arch OCI tarball -> ./result
nix build .#packages.aarch64-linux.image-<service>   # explicit arm64 build
nix run .#push <service> [tag]         # build BOTH arches + push multiarch

push uploads per-arch images as atcr.io/rtw.run/<service>:<tag>-amd64 and :<tag>-arm64, then creates manifest lists at :<tag> and :latest pointing at both. Run commands never change — the server pulls :latest and gets the right arch automatically.

aarch64 builds on an amd64 host (one-time setup) #

aarch64-linux derivations are built natively under QEMU user emulation (binfmt); nearly everything substitutes from cache.nixos.org, only waldo compiles locally.

  • NixOS: boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
  • Other distros (e.g. Arch/CachyOS): install qemu-user-static + qemu-user-static-binfmt and ensure systemd-binfmt registered the qemu-aarch64 handler (cat /proc/sys/fs/binfmt_misc/qemu-aarch64 → enabled, flags containing F). With a multi-user nix-daemon, also add extra-platforms = aarch64-linux to /etc/nix/nix.conf (keep any existing entries, e.g. i686-linux x86_64-v*-linux on CachyOS) and systemctl restart nix-daemon — the client-side --option extra-platforms that the push script passes only covers substitution, not local emulated builds.

Default tag is date-based: YYYY.MM.DDHHMM. Registry auth (one-time, interactive): podman login atcr.io -u rtw.run with an ATProto app password. In CI, provide the app password and use podman login atcr.io -u rtw.run --password-stdin.

Garbage collection (old releases) #

nix run .#gc <service> [keep]        # dry-run: what would be deleted
nix run .#gc -- <service> [keep] -y  # actually delete

Keeps :latest + the newest keep (default 3) dated releases; each expired release is removed as a triplet (<tag>, <tag>-amd64, <tag>-arm64) plus any manifest no longer referenced by a remaining tag.

Upstream updates #

nix run .#update -- --check          # drift report vs upstream (read-only)
nix run .#update -- update <svc>     # bump one service to its latest and rebuild

--check compares each image against its upstream and prints a table. update <svc> bumps the pinned version, regenerates the fixed-output hashes (source, cargo/vendor/npm-deps) via the Nix "fake hash → got:" loop, and rebuilds the host-arch image — it never pushes; review locally then nix run .#push <svc>. On-demand only (no timer/cron).

Which upstream each image tracks is declared in nix/updater.nix:

Service Strategy Source
pds npm @atproto/pds package (bluesky-social/atproto)
y-sweet github jamsocket/y-sweet release tag
waldo flake-input git.kilimanjaro.io/waldo input rev
pmtiles / graphhopper / caddy nixpkgs pkgs.* (advisory only — update via the nixpkgs input)

atcr.io does not implement the OCI delete endpoint (skopeo delete fails with "unsupported"): tags/manifests are ATProto records in the repo owner's PDS (io.atcr.tag / io.atcr.manifest), so gc deletes them via goat and the registry reflects the change within seconds. Executing (-y) requires a one-time goat account login -u rtw.run -p '<ATProto app password>' (dry-runs are read-only and need no auth). Blob storage reclamation is server-side (hold GC) — deleting manifests is all we can do client-side.

Registry page READMEs #

The README shown on each repo's atcr.io page (e.g. https://atcr.io/r/rtw.run/waldo) is an io.atcr.repo.page record in the owner's PDS (rkey = repo name) — it does not have to be set via the web UI. The markdown lives in docs/registry/<service>.md; publish with:

nix run .#readme <service>        # dry-run: prints the record JSON
nix run .#readme -- <service> -y  # create/update the record

Like gc, writing requires the one-time goat account login. The script sets userEdited: true so image pushes don't overwrite the page with manifest annotations, and preserves any avatar set via the UI when updating.

Service Image attr
graphhopper image-graphhopper
graphhopper-builder image-graphhopper-builder
waldo image-waldo
pmtiles image-pmtiles
y-sweet image-y-sweet
pds image-pds
caddy image-caddy

Production runbook #

Everything below lives outside the images and must be set up on the NixOS server by hand (or by the server NixOS config).

R2 credentials (rclone) #

Images that talk to R2 (graphhopper, graphhopper-builder) expect an rclone remote named r2. Two ways to provide it:

  1. Mounted config (recommended): write /etc/rclone/rclone.conf on the host with an [r2] remote (type s3, provider Cloudflare, access_key_id, secret_access_key, endpoint = https://<accountid>.r2.cloudflarestorage.com), mode 0600, and mount it into the container: -v /etc/rclone/rclone.conf:/config/rclone/rclone.conf:ro (rclone reads $XDG_CONFIG_HOME/rclone/rclone.conf; set XDG_CONFIG_HOME=/config or use --config /config/rclone/rclone.conf).

  2. Environment variables in the systemd unit (no file on disk):

    Environment=RCLONE_CONFIG_R2_TYPE=s3
    Environment=RCLONE_CONFIG_R2_PROVIDER=Cloudflare
    Environment=RCLONE_CONFIG_R2_ACCESS_KEY_ID=<key>
    Environment=RCLONE_CONFIG_R2_SECRET_ACCESS_KEY=<secret>
    Environment=RCLONE_CONFIG_R2_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
    

    (Plain AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY also work when the remote is configured with env_auth=true.)

Volumes and users #

Containers run as uid/gid 1000 (user svc). With rootless podman, add --userns=keep-id to the run command so container uid 1000 maps to your host user — otherwise container uid 1000 maps to a subuid and writes to bind-mounted host dirs fail with permission denied. The examples below assume --userns=keep-id and host dirs owned by your user. (With rootful/system podman instead: chown -R 1000:1000 the host dirs and drop the flag.)

Volume (in-container) Used by Backup?
/data/graphhopper graphhopper, graphhopper-builder No — re-downloadable/rebuildable cache
/data/osm graphhopper-builder (scratch) No
/var/log/graphhopper both Optional
/data waldo (world.parquet + .etag) No — re-downloadable
/data pmtiles (world.pmtiles symlink + daily builds) No — re-downloadable
/data y-sweet (local-store fallback when S3_URL unset) Yes — user documents
/data pds (sqlite DBs + blobs) Yes — all ATProto account data

graphhopper (routing server) #

Serves the GraphHopper routing API on port 8989 from a pre-built graph-cache on a volume.

Server setup #

mkdir -p /srv/rtw/graphhopper

Environment #

Var Default Meaning
GRAPH_CACHE_URL unset r2://bucket/path/graph-cache.tar.zst. If set, the graph is downloaded+extracted whenever the remote .md5 sidecar changes. If unset, /data/graphhopper/graph-cache must already exist (same-host builder case).
JAVA_OPTS -Xms16g -Xmx48g Server heap. MMAP_STORE (default) keeps the graph in page cache; raise heap only with RAM_STORE.

Example quadlet / podman run #

podman run -d --name graphhopper --userns=keep-id \
  -p 127.0.0.1:8989:8989 \
  -v /srv/rtw/graphhopper:/data/graphhopper \
  -v /etc/rclone/rclone.conf:/config/rclone/rclone.conf:ro \
  -e XDG_CONFIG_HOME=/config \
  -e GRAPH_CACHE_URL=r2://rtw-data/graphhopper/graph-cache.tar.zst \
  atcr.io/rtw.run/graphhopper:latest

Health check: curl http://localhost:8989/health.


graphhopper-builder (one-shot graph build job) #

Downloads Geofabrik regions, merges with osmium, imports a fresh graph-cache, packs graph-cache.tar.zst (+ .md5) and optionally uploads to R2. Run ad-hoc (e.g. monthly for fresh OSM data), not as a long-running service.

Sizing (full 6-region build) #

  • Disk: ~150GB scratch for PBFs + merged, then ~100GB graph — provision 400GB
  • RAM: works on 64–96GB with the default MMAP_STORE and -Xmx48g
  • Regions (default, ~71.5GB download): asia, australia-oceania, central-america, europe, north-america, south-america (Africa excluded to reduce memory; override with REGIONS)

Environment #

Var Default Meaning
REGIONS 6 continents minus Africa Geofabrik regions to download. Sub-regions work: REGIONS="asia/india asia/nepal" (files stored flat by basename).
GEOFABRIK_BASE https://download.geofabrik.de Mirror base URL
JAVA_OPTS_BUILD -Xms16g -Xmx48g Import heap
GRAPH_CACHE_URL unset r2://... upload target; unset = skip upload (graph stays on the volume for same-host serving)
KEEP_PBF 0 1 keeps downloaded/merged PBFs (debugging)
SKIP_DOWNLOAD 0 1 skips download+merge and imports from an existing /data/osm/merged.osm.pbf (needs a prior KEEP_PBF=1 run)

Example run (same-host, no upload) #

mkdir -p /srv/rtw/graphhopper /srv/rtw/graphhopper-build

podman run --rm --name graphhopper-builder --userns=keep-id \
  -v /srv/rtw/graphhopper-build:/data/osm \
  -v /srv/rtw/graphhopper:/data/graphhopper \
  atcr.io/rtw.run/graphhopper-builder:latest

For a small test build: add -e REGIONS=australia-oceania.

Example run (with R2 upload) #

Add rclone credentials (see above) and:

  -e GRAPH_CACHE_URL=r2://rtw-data/graphhopper/graph-cache.tar.zst

Testing different routing profiles #

To iterate on the config (profiles, custom models, encoded values) without re-downloading every time, keep the merged PBF from a first build, then rebuild with SKIP_DOWNLOAD=1:

# First run: download nepal + keep merged.osm.pbf
podman run --rm --userns=keep-id \
  -e REGIONS="asia/nepal" \
  -e KEEP_PBF=1 \
  -e JAVA_OPTS_BUILD="-Xms2g -Xmx8g" \
  -v /srv/rtw/graphhopper-build:/data/osm \
  -v /srv/rtw/graphhopper:/data/graphhopper \
  atcr.io/rtw.run/graphhopper-builder:latest

# Iteration runs: mount a tweaked config and skip the download+merge.
# The graph-cache is always wiped and re-imported from merged.osm.pbf.
podman run --rm --userns=keep-id \
  -e SKIP_DOWNLOAD=1 -e KEEP_PBF=1 \
  -e JAVA_OPTS_BUILD="-Xms2g -Xmx8g" \
  -e GRAPHHOPPER_CONFIG=/opt/graphhopper/custom.yml \
  -v $PWD/my-config.yml:/opt/graphhopper/custom.yml:ro \
  -v /srv/rtw/graphhopper-build:/data/osm \
  -v /srv/rtw/graphhopper:/data/graphhopper \
  atcr.io/rtw.run/graphhopper-builder:latest

Notes:

  • GRAPHHOPPER_CONFIG defaults to the baked-in /opt/graphhopper/config.yml; mount over it (or point the var elsewhere) to test profile changes.
  • Keep passing KEEP_PBF=1 on iteration runs or the merged PBF is deleted after import.
  • SKIP_DOWNLOAD=1 fails fast with a clear error if merged.osm.pbf is missing.

waldo (geocoder) #

Forward/reverse geocoder (WhosOnFirst admin areas only — cities, regions, countries; no addresses) on port 8999. The image contains the binary + entrypoint only; the parquet lives on the volume and is fetched from data.geocode.earth at container start (no R2, no rclone in this image).

Server setup #

mkdir -p /srv/rtw/waldo

Disk: the planet parquet is ~5.9GB and is replaced via .tmp + rename, so keep >= 12GB free on the volume.

Environment #

Var Default Meaning
PARQUET_URL https://data.geocode.earth/wof/dist/parquet/whosonfirst-data-admin-latest.parquet Source parquet. Country extracts work for small deployments/tests (e.g. .../whosonfirst-data-admin-ad-latest.parquet).
UPDATE_CHECK 1 1 = on start, if a local parquet exists, conditional-GET with the saved ETag and re-download when upstream changed. A failed check with a valid local file logs a warning and serves the stale copy. 0 = never touch the network when a local file exists.
DATA_DIR /data Where world.parquet and world.parquet.etag live.

First start downloads the full parquet (~5.9GB planet). /health only comes up after the parquet finishes loading into the in-memory DuckDB — allow a generous health-check start period (~1min planet-scale).

RAM: <2GB serving; ~2x transiently during a SIGHUP reload (replace world.parquet, then podman kill -s HUP waldo).

Example quadlet / podman run #

podman run -d --name waldo --userns=keep-id \
  -p 127.0.0.1:8999:8999 \
  -v /srv/rtw/waldo:/data \
  atcr.io/rtw.run/waldo:latest

Health check: curl http://localhost:8999/health. Forward geocode: curl 'http://localhost:8999/forward?q=kathmandu'.


y-sweet (realtime document store) #

Serves the y-sweet CRDT document persistence API (WebSocket + REST) on port 9000. The RTW app proxies connections to it upstream and handles all authn/authz, so y-sweet runs in development mode (no auth token, no --prod): the server must NOT be reachable from outside the network.

Persistence defaults to RTW's Cloudflare R2 account. With S3_URL unset the store is a local filesystem directory (/data — mount a volume there for rootless podman).

Environment #

Var Default Meaning
S3_URL unset s3://bucket[/prefix] persistence URL. Unset = local filesystem store in /data; set = S3-compatible bucket (R2 by default).
HOST 0.0.0.0 Listening IP address.
PORT 9000 Listening port.
CHECKPOINT_FREQ_SECONDS 10 How often document updates are checkpointed to the store.
AWS_ACCESS_KEY_ID – R2 API token access key. Required when S3_URL is set.
AWS_SECRET_ACCESS_KEY – R2 API token secret. Required when S3_URL is set.
AWS_ENDPOINT_URL_S3 https://3a6cd35b98a3708a39ad971cbd3df4e0.r2.cloudflarestorage.com S3-compatible endpoint. Override for other backends.
AWS_REGION auto Region (R2 ignores it; override for AWS).
AWS_S3_USE_PATH_STYLE true R2 requires path-style URLs. Set false for AWS.
AWS_SESSION_TOKEN unset Optional temporary-credential token.

Example run — R2-backed (production) #

Endpoint, region, and path-style are already defaulted to RTW's R2 account; only the bucket (inside S3_URL) and the API token secrets need setting — put the secrets in the systemd unit, not the run command:

podman run -d --name y-sweet --userns=keep-id \
  -p 127.0.0.1:9000:9000 \
  -e S3_URL=s3://rtw-data/ysweet \
  -e AWS_ACCESS_KEY_ID=<r2 access key> \
  -e AWS_SECRET_ACCESS_KEY=<r2 secret> \
  atcr.io/rtw.run/y-sweet:latest

No volume needed — documents persist in R2. S3_URL composes as s3://<bucket>/<prefix>: the host is the R2 bucket, the path is an optional folder inside it (e.g. s3://rtw-data/ysweet stores documents under the ysweet/ prefix). The R2 API token is created in the Cloudflare dashboard (R2 → Manage API Tokens, Object Read & Write).

Example run — local volume (test/standalone) #

mkdir -p /srv/rtw/y-sweet

podman run -d --name y-sweet --userns=keep-id \
  -p 127.0.0.1:9000:9000 \
  -v /srv/rtw/y-sweet:/data \
  atcr.io/rtw.run/y-sweet:latest

Health check: curl http://localhost:9000/ready.


pmtiles (tile server) #

Serves vector map tiles from a directory of .pmtiles archives on port 9090 at /{archive}/{z}/{x}/{y}.mvt (.mvt because the basemap tiles are MVT; pmtiles adds the extension from the archive's tile type). If the /data volume is empty it downloads the previous day's Protomaps planet build (~137GB, no API key) from build.protomaps.com, exposes it as https://tiles.rtw.run/world/{z}/{x}/{y}.mvt via a world.pmtiles symlink, and refreshes it on SIGHUP. Any archives you drop into /data yourself are served at /name/... instead and are never touched by the world update.

Server setup #

mkdir -p /srv/rtw/pmtiles

Disk: the daily planet build is ~137GB and is replaced on SIGHUP, so keep >= 300GB free on the volume.

Environment #

Var Default Meaning
PORT 9090 HTTP port
CORS http://localhost:5173,https://rtw.run,https://next.rtw.run Allowed origins for the Access-Control-Allow-Origin header
CACHE_SIZE 128 Header/directory LRU cache size in MB
PUBLIC_URL https://tiles.rtw.run Public base URL used for TileJSON

Example quadlet / podman run #

podman run -d --name pmtiles --userns=keep-id \
  -p 127.0.0.1:9090:9090 \
  -v /srv/rtw/pmtiles:/data \
  atcr.io/rtw.run/pmtiles:latest

Tile URL: https://tiles.rtw.run/world/{z}/{x}/{y}.mvt. The server is ready as soon as it starts (tiles are read lazily), so a health check can just hit a tile or the TileJSON (curl -fsS http://localhost:9090/world.json).

Refresh the planet: podman kill -s HUP pmtiles — downloads the newest daily build, atomically re-points world.pmtiles, restarts the server, and deletes the stale build. The daily file stays reachable at https://tiles.rtw.run/YYYYMMDD/{z}/{x}/{y}.mvt.


pds (ATProto personal data server) #

The Bluesky reference PDS (@atproto/pds), serving ATProto accounts and federating publicly via the bsky.network relay. Runs on port 3000.

The PDS service is hosted on its own second-level domain rtw.social — deliberately separate from the app's rtw.run domain, because the PDS serves both user-uploaded blobs and OAuth/session pages (see the atproto going-to-production guidance). User handles live on rtw.run (e.g. <user>.rtw.run) and are resolved to DIDs via https://<user>.rtw.run/.well-known/atproto-did (see PDS_SERVICE_HANDLE_DOMAINS below).

All configuration comes from PDS_* environment variables; the PDS never generates or persists secrets, and this image ships none. The container runs as uid 1000 and persists everything under /data (mount a volume there).

Secrets #

Three secrets are required in production — put them in the systemd unit (e.g. EnvironmentFile=), never in the run command:

Var Example generation
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32
PDS_JWT_SECRET openssl rand --hex 16
PDS_ADMIN_PASSWORD openssl rand --hex 16

ENV=development (set by devenv for local containers) generates throwaway secrets at startup instead and logs a warning — the instance is ephemeral and its data is not recoverable. In production ENV is unset and the entrypoint fails fast if any secret is missing.

Environment #

Var Default Meaning
PDS_HOSTNAME localhost (dev) Fully-qualified hostname; required in production
PDS_DATA_DIRECTORY /data Data dir on the volume (sqlite DBs)
PDS_BLOBSTORE_DISK_LOCATION $PDS_DATA_DIRECTORY/blobs Uploaded blobs (disk store)
PDS_PORT 3000 Listening port
PDS_CRAWLERS every relay on atproto.at/relays Comma-separated relay https:// URLs to notify of repo updates (via com.atproto.sync.requestCrawl). Defaults to the full relay directory to diversify away from Bluesky's stack.
PDS_INVITE_REQUIRED true Require an invite code to create an account (defaults on to prevent open signup/spam).
PDS_SERVICE_DID did:web:<hostname> Service DID override
PDS_SERVICE_HANDLE_DOMAINS .rtw.run,rtw.social Comma-separated handle domains the PDS serves /.well-known/atproto-did for. Leading dot = suffix (wildcard) match, so https://<user>.rtw.run/.well-known/atproto-did resolves any hosted <user>.rtw.run handle.
PDS_DEV_MODE false true relaxes the https-only URL requirement and disables SSRF protection (auto-set in dev)

OAuth page branding #

Defaults for these are baked into the image via the entrypoint script (nix/scripts/serve-pds.sh) and can be overridden per-deployment with -e / EnvironmentFile=. Color format is #rrggbb or rgb(r,g,b) — no alpha channel; only PDS_PRIMARY_COLOR gets an auto-computed foreground (black/white by WCAG contrast).

Var Default Meaning
PDS_SERVICE_NAME RTW Site name shown on the OAuth/login pages
PDS_HOME_URL https://rtw.run Footer "Home" link
PDS_LOGO_URL unset Logo image URL
PDS_PRIMARY_COLOR unset Primary accent color
PDS_ERROR_COLOR / PDS_WARNING_COLOR / PDS_INFO_COLOR / PDS_SUCCESS_COLOR unset Status/alert colors
PDS_BACKGROUND_LIGHT_URL / PDS_BACKGROUND_DARK_URL unset Light/dark background image URLs
PDS_TERMS_OF_SERVICE_URL / PDS_PRIVACY_POLICY_URL / PDS_SUPPORT_URL unset Footer links
PDS_CONTACT_EMAIL_ADDRESS unset Contact email

Admin access is HTTP basic auth with user admin and PDS_ADMIN_PASSWORD (e.g. for com.atproto.server.createInviteCode).

Example quadlet / podman run #

mkdir -p /srv/rtw/pds

podman run -d --name pds --userns=keep-id \
  -p 127.0.0.1:3000:3000 \
  -v /srv/rtw/pds:/data \
  -e PDS_HOSTNAME=rtw.social \
  -e PDS_SERVICE_HANDLE_DOMAINS=.rtw.run,rtw.social \
  -e PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX=<64-hex> \
  -e PDS_JWT_SECRET=<secret> \
  -e PDS_ADMIN_PASSWORD=<secret> \
  atcr.io/rtw.run/pds:latest

Health check: curl http://localhost:3000/xrpc/_health.

Source: bluesky-social/atproto · Docs: https://atproto.com/guides/self-hosting


caddy (reverse proxy) #

Caddy reverse proxy fronting every public RTW service, with automatic TLS (Let's Encrypt via Cloudflare DNS-01 — the image ships the caddy-dns/cloudflare module so it can issue the *.rtw.run wildcard cert). Runs on ports 80 and 443.

Two Caddyfiles are baked into the image (/etc/caddy/); the entrypoint picks one via RTW_ENV (dev or prod, default prod). Config is fully reproducible from the flake — nothing is hand-edited inside the container.

Route (prod) Upstream Notes
rtw.run 127.0.0.1:8080 RTW app; http → https redirect is automatic
geocoder.rtw.run 127.0.0.1:8999 waldo; CORS for *.rtw.run + http://localhost:5173
route.rtw.run 127.0.0.1:8989 graphhopper; CORS for *.rtw.run + http://localhost:5173
tiles.rtw.run 127.0.0.1:9090 pmtiles (sets its own CORS)
rtw.social 127.0.0.1:3000 PDS (XRPC, blobs, OAuth)
*.rtw.run 127.0.0.1:3000 PDS handle/DID resolution (/.well-known/atproto-did); specific subdomains above take precedence

y-sweet is not proxied — it is an internal service reached only by the RTW app.

Server setup #

mkdir -p /srv/rtw/caddy

The container runs as uid 1000, so binding to 80/443 needs the NET_BIND_SERVICE capability (--cap-add NET_BIND_SERVICE). ACME certs are persisted under /data (mounted below) so they survive restarts.

Environment #

Var Default Meaning
RTW_ENV prod dev uses the http-only *.localhost Caddyfile; prod uses the TLS one.
CF_API_TOKEN unset Cloudflare API token for ACME DNS-01 (prod; used to issue the *.rtw.run wildcard cert).

Example quadlet / podman run #

podman run -d --name caddy --userns=keep-id \
  --cap-add NET_BIND_SERVICE \
  -p 80:80 -p 443:443 \
  -v /srv/rtw/caddy:/data \
  -e RTW_ENV=prod \
  -e CF_API_TOKEN=<cloudflare api token> \
  atcr.io/rtw.run/caddy:latest

Local development #

Run the same image with RTW_ENV=dev (http only, no token needed). All routes are on *.localhost domains (rtw.localhost, geocoder.localhost, route.localhost, tiles.localhost, pds.localhost), which resolve to 127.0.0.1 on every modern system per RFC 6761 — no /etc/hosts entries or local DNS required.

Source: Caddy · caddy-dns/cloudflare