spindle
Self-hosted Tangled spindle
Star 0
Fork 0
Watch
spindle docs runbook.md
24 kB · 420 lines
Markdown
at main

Spindle runbook #

Commands run from this checkout on the owner's Mac, unless a step names another location. Only the lead operates deployed systems. Steps marked (not yet drilled) need live verification.

Architecture and state #

knot (tangled.org) --events--> mill (Railway, spindle.demi.dev)
                                  ^  |
                                  |  | websocket /mill
                                  v  v
                             executor (home NixOS VM, nixery on Docker)
                                  |
                                  v
                             Railway S3 bucket
                             (also read by the mill)

The executor dials wss://spindle.demi.dev/mill. It needs no inbound internet ports.

Piece State and location
Mill Volume mill-data at /data: spindle.db, spindle.db-wal, spindle.db-shm, spindle.db.lock, and hydrant/. The mill creates repos/ and logs/ on demand.
Identity Owner's Mac: ~/.config/spindle/spindle-key.pem. Password manager: backup. Railway: SPINDLE_SIGNING_KEY_PEM. The entrypoint writes /run/spindle/key.pem. This runtime copy is not a backup.
Executor VM spindle-exec-1: /var/lib/spindle/ holds SQLite, the event outbox and local job state. /var/lib/secrets/ holds the token and bucket credentials. The Nix store contains no secrets.
Bucket Railway resource spindle-artifacts, region iad: workflow artifacts and logs. The actual S3 bucket name comes from the BUCKET reference variable.

Operator context #

Replace the placeholders in commands with these values:

Placeholder or resource Value
<project> 5caa03b7-45a1-4867-9147-3d7325a53fc6 (project spindle)
<env> 7b5decec-ce78-45f0-b6c0-996bb87323a1 (production)
<mill> e07b46fb-a540-4bed-a051-5760517043d4 (service mill)
Executor SSH ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94
Current registration home-1-rotation-1, labels linux and x86_64, 4 seats
Railway domain mill-production-da13.up.railway.app
Custom domain spindle.demi.dev. Namecheap CNAME spindle → 1hbtrau7.up.railway.app, TXT _railway-verify.spindle

Do not touch other VMs or containers on the Unraid host (root@192.168.1.164). Keep PORT=6555 and RAILWAY_DOCKERFILE_PATH=mill/Dockerfile in Railway. Railway healthchecks use PORT.

Run every mill CLI command as uid 10001 (spindle), with --db /data/spindle.db:

setpriv --reuid=spindle --regid=spindle --init-groups -- spindle mill executor list --db /data/spindle.db

Root can create files that the mill cannot use. The entrypoint changes ownership of /data, not its existing contents.

Health and connection checks #

  1. Check the mill endpoints:

    curl https://spindle.demi.dev/_health
curl https://spindle.demi.dev/.well-known/did.json

    Expected: HTTP 200 and DID did:web:spindle.demi.dev.

  2. List the registrations:

    railway ssh -p <project> -e <env> -s <mill> -- 'setpriv --reuid=spindle --regid=spindle --init-groups -- spindle mill executor list --db /data/spindle.db'

    executor list shows NAME CREATED EXPIRES LABELS, not live connections. It never shows tokens or hashes.

  3. Check the mill service logs in Railway for a fresh executor connected node=home-1-rotation-1 or executor reconnected node=home-1-rotation-1. After rotation, use the new registration name instead of home-1-rotation-1.

  4. Check the VM journal:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'journalctl -u spindle | grep "connected to mill"'

    The VM log uses its configured hostname, spindle-exec-1, not the mill registration name. Check the timestamp after the operation.

  5. Run one pipeline on demi.dev/pneumo from its Pipelines page. Check that it finishes green and its logs and artifacts load.

Upgrade the pinned Tangled version #

Mill and executor must run the same version. The current pin is v1.20.0 (2096d06d).

  1. Bump the tag in mill/Dockerfile and executor/flake.nix in one PR.

  2. Regenerate executor/flake.lock with mise run executor:lock.

  3. Run mise run check and mise run mill:build before deployment.

  4. Wait for running pipelines to finish before deployment.

  5. Deploy the mill from this checkout:

    railway up --detach -p <project> -e <env> -s <mill>

    Railway builds mill/Dockerfile. The first cold build took about 19 minutes. Cached builds took less than one minute.

  6. Copy the executor configuration to the VM:

    rsync -a --delete -e "ssh -i ~/.ssh/spindle_executor_ed25519" executor/ root@192.168.1.94:/root/spindle/executor/

    --delete applies only to /root/spindle/executor/.

  7. Build the executor system. Replace <nixpkgs rev from executor/flake.lock> with nodes.nixpkgs.locked.rev from that file.

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'cd /root/spindle && nix shell "github:NixOS/nixpkgs/<nixpkgs rev from executor/flake.lock>#git" -c nix build --no-update-lock-file --out-link /root/spindle/result-toplevel "path:/root/spindle/executor#nixosConfigurations.executor.config.system.build.toplevel"'

    The VM has no git. The temporary shell supplies it for the git+https input. The first source build took about 15 minutes.

  8. Switch the executor system with the proven detached command:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'P=$(readlink -f /root/spindle/result-toplevel); systemd-run --unit=spindle-switch --collect --property=Type=oneshot --setenv=PATH=/run/current-system/sw/bin:/run/wrappers/bin /run/current-system/sw/bin/sh -c "nix-env -p /nix/var/nix/profiles/system --set $P && /nix/var/nix/profiles/system/bin/switch-to-configuration switch > /root/spindle/switch.log 2>&1; echo exit=\$? >> /root/spindle/switch.log"'

    This performs the build-and-switch operation normally handled by nixos-rebuild switch. No direct nixos-rebuild command is proven here. systemd-run protects the switch from SSH disconnection during network changes.

  9. Read /root/spindle/switch.log on the VM. Check that its final line is exit=0.

  10. Complete all health and connection checks, including one green pipeline.

Executor tokens #

Never use shell tracing (set -x), terminal copies, command arguments or logs for secret values. token generate stores a hash in SQLite and writes the raw token to stdout. Always redirect or pipe that output. railway ssh writes its banner to stderr, so stdout can carry a token directly to the VM.

Generate, add and deliver (proven in ENG-468) #

ENG-468 used home-1 for the initial registration. That name is now revoked; do not reuse it. For an existing registration, use the rotation procedure instead.

  1. Disable tracing and enable pipeline failure detection in the local Bash shell:

    set +x
set -o pipefail

  2. Generate, register and deliver the token without terminal output:

    railway ssh -p <project> -e <env> -s <mill> -- 'set -eu; t=/run/spindle/home-1.token; as() { setpriv --reuid=spindle --regid=spindle --init-groups -- "$@"; }; as sh -c "umask 077; spindle mill executor token generate --db /data/spindle.db > $t"; as spindle mill executor add home-1 --db /data/spindle.db --token-file "$t" --label linux --label x86_64 >&2; cat "$t"; rm -f "$t"' 2>/dev/null \
  | ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'umask 077; install -d -m 700 /var/lib/secrets; cat > /var/lib/secrets/spindle-executor.token; chmod 600 /var/lib/secrets/spindle-executor.token; wc -c < /var/lib/secrets/spindle-executor.token'

    Output is only the byte count. If the pipeline fails or the count is zero, stop before restarting the executor. The command suppresses Railway stderr, including registration errors. A failed registration must not count as successful delivery.

  3. Restart the executor after successful delivery:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'systemctl restart spindle'

  4. Complete the health and connection checks.

token generate has no expiry by default. --ttl accepts a duration, for example 720h. add accepts only a token that token generate stored. Reusing a name with another token fails. It does not replace the hash. Reusing the same name and token updates its labels.

Rotate home-1-rotation-1 (drilled 2026-11-01, ENG-471) #

ENG-471 drilled home-1 → home-1-rotation-1 on 2026-11-01, 17:29–17:34 UTC; pipeline 3mzbdmqmve522 was green; home-1 was revoked.

This procedure saves the old token on the VM before delivery. The old registration stays valid until the new executor reconnects. The example rotates home-1-rotation-1 to the new, unused name home-1-rotation-2. It does not change the VM hostname or Nix configuration. The mill derives the identity from the token. The executor does not send its configured name in the handshake.

CAUTION: Do not reset the stream during a pipeline. The reset discards unacknowledged events and pending artifact records. A new identity has no stream cursor. Reusing the old stream can cause gap in stream seqnos errors.

  1. Pause new pipeline submissions and wait for running pipelines and their log and artifact uploads to finish.

  2. Run executor list from health and connection checks. Check that home-1-rotation-1 exists and home-1-rotation-2 does not. For later rotations, substitute the current name and a fresh, unused name throughout this procedure.

  3. Disable tracing and enable pipeline failure detection with the two shell commands from the initial delivery procedure. If a .prev file remains from an earlier attempt, complete rollback before continuing. Save the old token before step 4:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'cp -p /var/lib/secrets/spindle-executor.token /var/lib/secrets/spindle-executor.token.prev'

    If the copy fails, stop before step 4.

  4. Generate, register and deliver a new token. This adapts the proven delivery command to the new name:

    railway ssh -p <project> -e <env> -s <mill> -- 'set -eu; t=/run/spindle/home-1-rotation-2.token; as() { setpriv --reuid=spindle --regid=spindle --init-groups -- "$@"; }; as sh -c "umask 077; spindle mill executor token generate --db /data/spindle.db > $t"; as spindle mill executor add home-1-rotation-2 --db /data/spindle.db --token-file "$t" --label linux --label x86_64 >&2; cat "$t"; rm -f "$t"' 2>/dev/null \
  | ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'umask 077; install -d -m 700 /var/lib/secrets; cat > /var/lib/secrets/spindle-executor.token; chmod 600 /var/lib/secrets/spindle-executor.token; wc -c < /var/lib/secrets/spindle-executor.token'

    The expected byte count is 44: 43 base64url characters plus a newline. If the pipeline fails or the count differs, complete rollback before retrying. Do not revoke the old registration. The saved .prev file preserves its token even if the executor restarts.

  5. Stop the idle executor, reset its stream, and restart it:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'systemctl stop spindle && spindle executor reset-stream --db /var/lib/spindle/spindle.db && systemctl restart spindle'

    If the command fails, complete rollback before retrying.

  6. Check for a fresh mill log entry executor connected node=home-1-rotation-2 and a fresh VM connected to mill entry. If either fresh entry is absent, complete rollback before revoking the old token.

  7. Run one pipeline and check that it finishes green, with readable logs and artifacts.

  8. Revoke the old token by its registration name:

    railway ssh -p <project> -e <env> -s <mill> -- 'setpriv --reuid=spindle --regid=spindle --init-groups -- spindle mill executor revoke home-1-rotation-1 --db /data/spindle.db'

    revoke deletes the token hash and its registration. It rejects future connections. It does not close an existing websocket.

  9. Run executor list again. Check that only the new registration remains for this VM. After the old registration is absent, remove the saved token:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'rm /var/lib/secrets/spindle-executor.token.prev'

  10. Resume pipeline submissions and record the drill evidence in the issue.

Evidence: UTC timestamps, registration lists before and after, fresh connection log entries, the green pipeline URL, and the revoke exit status. Do not include tokens, token hashes, environment values or secret-file contents.

Roll back a failed rotation (not yet drilled) #

Do not revoke home-1-rotation-1. The saved token still authenticates as the old registration.

  1. Restore the old token file without printing its contents:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'mv /var/lib/secrets/spindle-executor.token.prev /var/lib/secrets/spindle-executor.token'

  2. If step 5 stopped or restarted the executor, wait until all pipelines and their log and artifact uploads finish. Stop the idle executor, reset its stream, and restart it after restoring the file:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'systemctl stop spindle && spindle executor reset-stream --db /var/lib/spindle/spindle.db && systemctl restart spindle'

    The fresh epoch starts sequence numbers at 1 for the old identity. Check for fresh connection logs under home-1-rotation-1 on the mill and connected to mill on the VM.

  3. Run executor list from health and connection checks. If home-1-rotation-2 exists, revoke that incomplete registration:

    railway ssh -p <project> -e <env> -s <mill> -- 'setpriv --reuid=spindle --regid=spindle --init-groups -- spindle mill executor revoke home-1-rotation-2 --db /data/spindle.db'

    If listing or revocation fails, stop and keep the restored token. Resolve the error before retrying.

  4. Check that home-1-rotation-1 exists and home-1-rotation-2 does not with executor list.

  5. Start again from step 2 of the rotation procedure. Step 3 creates a new .prev copy before delivery.

Bucket credentials #

/var/lib/secrets/ is root-owned, mode 700. Both secret files are root-owned, mode 600. spindle-executor.env contains AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, and SPINDLE_ARTIFACT_STORES_S3_BUCKET.

The ENG-468 delivery command copies the mill's bucket variables without terminal output:

railway ssh -p <project> -e <env> -s <mill> -- 'for k in AWS_ENDPOINT_URL AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_REGION SPINDLE_ARTIFACT_STORES_S3_BUCKET; do eval "v=\${$k-}"; printf "%s=%s\n" "$k" "$v"; done' 2>/dev/null \
  | ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'umask 077; cat > /var/lib/secrets/spindle-executor.env; chmod 600 /var/lib/secrets/spindle-executor.env'

After successful delivery, restart spindle. Never read the environment file into terminal output. The module's interactive spindle wrapper overrides the bucket with an empty value (ENG-482). For a manual server run, set the bucket explicitly without printing credentials. The DB-only executor reset-stream command does not start the server.

Signing key #

The private key lives at ~/.config/spindle/spindle-key.pem on the owner's Mac (file 600, directory 700). The owner keeps a backup in the password manager. Railway stores the key as SPINDLE_SIGNING_KEY_PEM, outside /data.

Current compressed public key (hex, safe to share):

02914161f399872c226091dfe526f977f86414d189f2ee8e30bb39696b8acf92a7

Restore the existing key (not yet drilled) #

  1. Restore the password-manager attachment to ~/.config/spindle/spindle-key.pem, without terminal output.

  2. Set the directory mode to 700 and the file mode to 600.

  3. Set the Railway variable through stdin:

    railway variable set SPINDLE_SIGNING_KEY_PEM --stdin --skip-deploys -s <mill> -e <env> -p <project> < ~/.config/spindle/spindle-key.pem
railway redeploy -y -p <project> -e <env> -s <mill>

  4. Complete the health and connection checks.

The same key preserves the identity. Volume backups do not replace this separate key backup.

Lost key (not yet drilled) #

  1. If all copies are lost, generate a replacement P-256 key into the protected file, not stdout:

    umask 077
mkdir -p ~/.config/spindle
chmod 700 ~/.config/spindle
openssl ecparam -name prime256v1 -genkey -noout -out ~/.config/spindle/spindle-key.pem
chmod 600 ~/.config/spindle/spindle-key.pem

  2. Save the replacement in the password manager.

  3. Set the Railway variable and redeploy with the two commands from the key restore procedure.

  4. Remove and re-add spindle.demi.dev at tangled.org/settings/spindles. Tangled v1.20.0 has no “verify” button. The list entry is the registration.

  5. Complete the health and connection checks.

  6. Record the new public key in this runbook.

A replacement key changes the spindle's signing identity, even though its hostname and DID remain the same.

Mill backups and restore #

SQLite uses WAL mode. Back up the whole /data volume, including its sidecars and hydrant/, not just spindle.db. Railway volume backups do not include variables, the signing-key backup, VM secret files or bucket objects.

Enable backups — lead, once (not yet drilled) #

  1. Open Railway project spindle → production → service mill → Backups.
  2. Check that the attached volume is mill-data, mounted at /data.
  3. Enable a daily backup schedule. Railway retains daily backups for six days.
  4. Create a manual backup before the first restore drill.
  5. Record the successful backup timestamp and schedule in the issue, without secret values.

Railway documents the UI, retention and restore behavior in Volume backups. Backup enablement and restore are not yet proven for this service.

Restore /data (not yet drilled) #

CAUTION: A restore replaces live state with the snapshot. Newer jobs, registrations and token revocations can be lost.

  1. Pause pipeline submissions and wait for running jobs to finish.

  2. Stop the executor with systemctl stop spindle on the VM.

  3. Select the backup timestamp in the mill service's Backups tab, in the same project and environment.

  4. Click Restore and review the staged changes under Details. Railway creates a new volume at the same mount path. It retains the old volume, unmounted, with its newer backups.

  5. Check that the restored volume mounts at /data, then click Deploy. The mill entrypoint must still run as uid 10001 after startup. The restore does not restore Railway variables.

  6. Inspect the restored file names and ownership without reading their contents:

    railway ssh -p <project> -e <env> -s <mill> -- 'ls -la /data'

  7. Check the endpoints and registrations from health and connection checks.

  8. Reconcile the registrations with rotations since the snapshot. Revoke credentials that the snapshot restored but the lead previously revoked. If the VM token postdates the snapshot, generate and deliver a fresh token under an unused registration name.

  9. Reset the stopped executor's stream, then restart it:

    ssh -i ~/.ssh/spindle_executor_ed25519 root@192.168.1.94 'spindle executor reset-stream --db /var/lib/spindle/spindle.db && systemctl restart spindle'

    The reset accepts loss of unacknowledged events and pending artifact records. It prevents gaps against the restored mill cursor. The source also exposes mill executor reset-cursor, not reset-acked. Clearing only the mill cursor cannot recover events the executor already removed.

  10. Complete all health and connection checks, including one green pipeline with readable logs and artifacts.

  11. Record the snapshot timestamp, recovery results and lost interval before resuming submissions.

Keep the original volume until recovery succeeds. Do not wipe it. Railway deletes all backups of a wiped volume. The bucket stays separate. A volume restore cannot recover missing bucket objects.

Outages #

Symptom Behavior and action
Home server, VM or home internet unavailable The mill queues up to 100 pending workflows. It rejects placement beyond that limit. Running jobs fail after the 45-second reconnect grace. Restore the VM or network, then check fresh connection logs and retry failed pipelines.
VM reboot ENG-468 observed reconnection in 24 seconds, without manual steps. Check the journal and a green pipeline after recovery.
Mill redeploy Executors reconnect automatically. Check for executor reconnected in the mill log. A redeploy that exceeds the recovery window can still fail running jobs.
Healthcheck fails after deployment Check PORT=6555, the Railway service logs, and the /data mount and ownership. A registration list alone does not prove that the executor is online.
Bucket error Check that both processes use the same bucket and that the VM environment file has mode 600. Do not print credential values.

For a long outage, switch a repo to hosted CI only after Tangled fixes the hosted quota:

  1. Open the repo's Settings → Pipelines.
  2. Select Hosted by Tangled.
  3. Run one nixery pipeline and check that it finishes green.

The current hosted quota is 4 GiB. Nixery requests 6 GiB. The fallback does not work while that mismatch remains.

LAN isolation is proven: job containers cannot reach 192.168.1.1, 192.168.1.164, or 192.168.1.94:22. Public tangled.org and nixery.tangled.sh returned 200. Container DNS uses only 1.1.1.1. ENG-474 tracks a second resolver.

Cost #

Railway dashboard → workspace stoopkidddd's Projects → Usage shows the usage charges. Expected cost is about $5/month for the mill service, volume and bucket. Bucket storage costs $0.015/GB-month. Volume backups add incremental storage charges. The executor uses existing home hardware.

Source basis #

ENG-465/466/468 supplied the proven deployment, delivery and recovery observations. Undrilled procedures use Tangled v1.20.0 source:

Source Operational rule
cmd/spindle/main.go token generate, add, list, revoke, reset-cursor, and executor reset-stream flags and effects
spindle/db/mill_tokens.go, spindle/db/db.go Same-name token replacement fails. Revocation deletes the hash and cascades to the registration.
spindle/mill/handler.go Token-derived executor identity. Authentication occurs before websocket upgrade.
spindle/mill/mill.go, spindle/mill/restore.go Stream sequence continuity, reconnect grace, queue limit and restored leases
spindle/config/config.go, nix/modules/spindle.nix Runtime state paths, credential loading, signing-key format and executor restart behavior