Tangled-compatible personal CI server based on Nix

ci DESIGN.md

41 kB · 712 lines

Markdown

BT · docs: deploys by hand and in CI share the runner's age identity · 1w ago

ci: trusted private CD on Nix — Design #

Status: built and running on infra.rtw.run. This describes the system as it stands; SECURITY.md has the security model.

ci runs trusted, private deploys. It runs workflows in each repo's devenv shell, gates deploys behind a decision a person can make from a phone, and shows public CI from Tangled's hosted spindle on the same board. It isn't a Tangled-compatible runner, and it doesn't run public repos' CI.

1. Goals and non-goals #

Goals

Manage the fleet from a phone: one board for deploys and public CI, so an urgent problem can be handled without a laptop.
Gated CD: a person approves deploys; health is reported, rollback is one button. Deploys can be started from a remote Claude session (/rc) without the dev machine.
Local parity: the exact same flow runs from the command line on the dev machine. No push-and-pray loop.
Generic: nothing in the engine knows about sovrn, moods, Colmena or NixOS deploys. Any project can reuse it.
Multi-arch (x86_64 + aarch64) from one runner, through Nix.

Non-goals (for now)

Running untrusted code. Only the owner's code runs here; isolation exists to keep CI from changing the host's config or competing with its services, not to contain an attacker. Public repos' CI runs in Tangled's VMs.
Being a spindle or a knot. Lexicons are shared only where that saves work.
Multiple runners, autoscaling, job claiming.
A polished UI.

2. Principles (the decisions behind this design) #

No programming in YAML. The workflow file is an inventory of actions plus the default order. All logic (what to show, what happens next, when to ask a human) lives in code — Nix, shell, Justfiles — and reaches the engine through a typed JSON result contract.
The engine is the product; the runner is a host for it. The same ci binary runs locally and on the runner, pinned per repo through devenv.
Steps are the happy path; actions are everything the workflow can do. A step's result can route to any declared action, including happy-path steps.
Gates are monotonic. Code can add a gate, never remove one.
Health is notify-only. Transient errors are common after a deploy; the operator decides whether to roll back.
Secrets are a repo. An encrypted vault cloned at a known path; the secrets tool behaves identically everywhere. The engine has no secrets concept.
Simplicity over isolation strength. Sandboxed systemd units, not VMs, until something needs more.

3. System overview #

 public repos                            private repos
 knot.rtw.run (Knot2, @rtw.run)          git@git.kilimanjaro.io:<repo> (gitolite)
      │  firehose: when to look               │
      ▼                                       │  fetched by the runner
 spindle.tangled.network (hosted)             ▼
      │  API: what happened             runner daemon ── sandboxed units ─► ci (engine)
      │                                       │          (ci.slice, user ci)  in devenv shell
      ▼                                       ▼
 appview ◄────────────────────────────── runner API (token)
    └─► browser / phone (owner), JSON + text API (agents: private runs only)

 dev machine:  `ci run <workflow>` — same engine, same manifest, same recipes

The URL says which side a repo is on: knot.rtw.run is public, git.kilimanjaro.io over SSH is private, and there are no redirects between them. git.kilimanjaro.io/<project> over HTTPS answers only Go's go-import lookups, pointing at the knot. All of this runs on infra.rtw.run, and the servers repo owns the forge's configuration.

Part	Where	Responsibility
Public CI	Tangled's hosted spindle	`engine: microvm` workflows in public repos, in `devenv shell` so `just test` matches local
Runner daemon + engine	`infra.rtw.run`	Run private workflows, hold the fleet's keys, deploy
Appview	`infra.rtw.run`	Show both kinds of run, gates and actions
Bridge	`infra.rtw.run`	Turn pushes to Gitolite, reported by a post-receive hook, into events for the runner (§5)

The bridge, runner daemon and appview are three roles of one Go binary.

4. Identity and records #

A private repo's identifier is did:web:<forge-host>:<repo> (e.g. did:web:git.kilimanjaro.io:servers). It's only a name inside this system; no DID document is served for it. Public repos have Tangled's did:plc DIDs, held in org.tangled.repo.declaration records on @rtw.run.
The runner keeps its records in runner.db for its own history; nothing reads them from outside. They use sh.tangled.pipeline, sh.tangled.pipeline.status and sh.tangled.git.refUpdate, whose shapes fitted when this system set out to be compatible with Tangled, plus our own run.rtw.ci.*:
- run.rtw.ci.workflow — the manifest schema (§6)
- run.rtw.ci.step.result — the step result contract (§7)
- run.rtw.ci.gate — {pipeline, workflow, action, options[], state: waiting|approved|rejected|expired|superseded, actor, prompt}
- run.rtw.ci.transition — engine routing decisions, for the timeline
- run.rtw.ci.event — one line of a run's events.ndjson
Manual runs start through sh.tangled.ci.triggerPipeline on the runner's API.
Public runs are read through org.tangled.ci.* on the spindle (§11).
Lexicons are the single schema source. Go types are hand-written for the handful of records used, and a test checks each Go struct against its lexicon's properties so they cannot drift.
Auth between services: a bearer token, on localhost.

5. Bridge (optional) #

Every push to a gitolite repo reaches the runner through the bridge, except gitolite-admin's, whether or not a workflow wants it. Deploys are manual, so they still start by hand; a workflow with event: [push] starts on a push:

services.ci.bridge.gitolite installs a post-receive hook for every repo, which runs ci bridge notify --spool <dir>. It reads git's <old> <new> <ref> lines on stdin, takes the repo and user from gitolite's GL_REPO and GL_USER (or --repo), computes each update from the bare repo (changed files, commits per author, default ref, push options), writes one JSON file per push into the spool, pokes the bridge, and always exits 0: it never blocks or fails a push.
The hook runs as git, which joins the ci-bridge group for the spool. It never gets the bridge's token, which can read /events. It can poke with --server and --token-file where that's wanted; the module's doesn't.
The bridge server ingests the spool on start, on every poke and every 2 s, and deletes each file once its events are in the log; unparsable files are moved aside (.bad). A bridge that is down only delays events.
It emits one sh.tangled.git.refUpdate per pushed ref on its token-protected /events, and the runner follows that from a cursor. -o skip-ci is honoured by the runner. The bridge doesn't read workflows.
Event log (internal/eventlog, shared with the runner): SQLite (created INTEGER PRIMARY KEY, rkey, nsid, event), where created is a monotonic UnixNano and the cursor.

6. Workflow manifest #

.tangled/workflows/<name>.yml. Pure data, validated against run.rtw.ci.workflow. No conditionals, expressions or loops.

# servers/.tangled/workflows/deploy-sovrn.yml, abridged
when:                                   # matched when a run is started
  - event: [manual]
engine: devenv                          # the runner skips any other engine
repos:                                  # extra checkouts, pinned per run
  - { repo: data, path: ../data, env: SECRETS_DIR }   # exports the path

steps: [plan, deploy, health-check, record]   # the happy path

actions:
  plan:         { command: just plan-project sovrn }
  deploy:       { command: just apply-project sovrn,
                  gate: { prompt: "Deploy sovrn?" },
                  concurrency: { group: fleet, supersede: true },
                  audited: true }
  health-check: { command: just prod-health-check sovrn }
  record:       { command: just record-deploy sovrn }
  rollback:     { command: just rollback, label: Roll back, style: danger,
                  manual: true, audited: true,
                  gate: { prompt: "Roll sovrn back to what ran before this deploy?" },
                  concurrency: { group: fleet } }

Both Tangled's spindle and this runner read .tangled/workflows, and they don't overlap. A spindle marks a workflow with an engine it doesn't have as failed, so engine: devenv workflows belong in private repos, where no spindle looks. The runner skips other engines, so a public repo's engine: microvm workflow is never its concern.

A smaller project needs only what it uses:

when: [{ event: [manual], branch: [main] }]
engine: devenv
steps: [test, deploy]
actions:
  test:   { command: devenv test }
  deploy: { command: just deploy, gate: { prompt: "Deploy?" } }

Action fields

Field	Meaning
`command`	Shell command, run as `devenv shell bash -euo pipefail -c '<command>'` (§9)
`dir`	Working directory relative to the repo checkout (default `.`)
`gate`	`{ prompt, timeout }` — always ask a human before running (default timeout 24h)
`concurrency`	Named lock; `{ group, supersede }` — supersede cancels older runs waiting at a gate for the same group
`manual`	Show as a button on the run page at any time
`style`	`normal` or `danger` (rendering only)
`input`	`{ label }` — ask for one line of text when answering this action's gate; requires `gate`. The action reads it from `$CI_ANSWER`
`audited`	Local runs that execute this action report to the appview (§10)

The set of actions is closed: results may only route to declared actions. That makes the manifest an honest list of what the workflow can do, and it is what the UI renders as buttons.

7. Step result contract #

stdout/stderr are logs. The result is written to the file at $CI_RESULT.

{
  "version": 1,
  "status": "failure",
  "summary": "mx99.eu.sovrn.at /healthz 503 for 4m (stalwart: connection refused)",
  "next": [
    { "action": "rollback", "prompt": "mx99 degraded after deploy. Roll back sovrn?", "then": "stop" },
    { "action": "health-check", "label": "Check again" }
  ],
  "gated": true,
  "attachments": [ { "label": "health probes", "path": "health.ndjson", "type": "ndjson" } ],
  "outputs": { "hosts": ["mx99.eu.sovrn.at"] }
}

Field	Meaning
`status`	`success` · `warning` · `failure`
`summary`	Markdown shown on the run page, at gates and in notifications
`next`	Omitted → default routing. `"stop"` → end the run. Otherwise one option or a list
`next[].action`	A declared action (happy-path step or not)
`next[].then`	After that action succeeds: `continue` (resume the happy path) or `stop`. Default: `continue` for happy-path steps, `stop` otherwise
`next[].prompt`, `label`, `inputs`	Text for the person; data passed to the action as `$CI_INPUTS`
`gated`	Ask a human before taking `next`
`attachments`	Files under `$CI_ARTIFACTS`, rendered by type (`ndjson`, `text`, `markdown`)
`outputs`	Merged into `$CI_STATE/outputs.json`

Fallbacks and errors

No result file: exit 0 → {status: success}; non-zero → {status: failure, next: "stop"}. Plain recipes work unchanged.
Invalid JSON, schema mismatch, an undeclared action, a path outside $CI_ARTIFACTS, or non-zero exit claiming success → treated as failure with the contract error shown.
Each step runs with a private /tmp; anything worth keeping goes in $CI_ARTIFACTS or $CI_STATE.

A ci-result helper on PATH writes valid results from shell:

ci-result fail --summary "mx99 /healthz 503 for 4m" \
  --next rollback --prompt "Roll back sovrn?" --then stop \
  --next health-check --label "Check again" \
  --attach "health probes=health.ndjson"

8. Routing semantics #

The run keeps a cursor into steps.

run action → read result
  ├─ next omitted  → success: happy-path action → next step; other action → stop
  │                   failure: stop
  ├─ next = "stop" → end run
  └─ next = [A, …] → gate if (any option's action declares gate) OR result.gated OR >1 option
                      gate offers each option + Dismiss (end run with this step's status)
                      no gate → run A
  naming a happy-path step moves the cursor there (forward skips, backward repeats)
  after A succeeds: then=continue → next step after cursor; then=stop → end run

Rules

Gates are monotonic: a declared gate can't be bypassed by a result. A choice between several options always needs a person.
Off-path actions end the run unless their then or their own result says otherwise. Nothing slides back onto the happy path by accident.
The chosen action receives $CI_TRIGGER_RESULT (the result that routed to it), $CI_INPUTS, and the shared $CI_STATE.
Loop guard: at most 50 transitions per run (configurable).
Final status, by how the run ends:
- it runs off the end of the happy path → success;
- a result says next: "stop" → success, or failure if that result is a failure (the way for an action to end a run early on purpose);
- an action's then: stop → failure if that action is off the happy path ("diverted: stopped after rollback"), even when the last step had already started; for a happy-path step, success only if it was the last one;
- a failure with no next → failure; a contract or engine error → failure; the loop guard → failure;
- dismissing a declared gate reached by default routing → cancelled; dismissing a gate raised by a result → that result's status (failure stays failure, success/warning → success); an expired gate, --until or Ctrl-C → cancelled. These are recorded as sh.tangled.pipeline.status (cancelled and failed).
If the runner has a notifier configured (--ntfy; none is set today), gates and runs that end unsuccessfully notify (runner config, not code).
Approving a gate or running a danger action requires the owner's logged-in session. The agent API token (used from /rc) can read, trigger and re-run, but not approve.

9. Engine (`ci`) #

One Go binary. Repos pin it as a devenv input (packages = [ inputs.ci… ]), so the dev machine and the runner use the identical engine for a given commit.

Commands

ci validate                         # manifest + actions resolvable; schemas
ci run <wf>                         # from the working copy; gates prompt in the terminal
ci run <wf> --clean [--rev @-]      # fresh worktree of a commit, as the runner does
ci run <wf> --until plan            # stop before an action: safe rehearsal
ci run <wf> --stub health-check=fixtures/degraded.json   # fake a result, test routing
ci run <wf> --detach                # exit at gates, as on the runner
ci action <name> --no-follow        # run one action, print result + what routing would do
ci resume <run> [--choose rollback|N] [--dismiss] [--expire]   # answer a paused run's gate
ci action rollback --run <run>      # manual action against an earlier run's state
ci view <run>                       # render events.ndjson
ci runs                             # list local runs ($CI_RUNS_DIR, default ~/.local/state/ci/runs)
ci trigger <repo> <wf> [--rev]      # ask the runner for a manual pipeline
ci wait <repo> <sha>                # block until done; print failing step's logs

Actions run as devenv shell bash -euo pipefail -c 'cd "$CI_ACTION_DIR"; <command>' started in the main checkout, so the workflow's repo pins the toolchain and the engine even when an action's dir is another repo (../servers). --shell bash skips devenv, for debugging only. Exit codes: 0 success, 1 failure, 2 usage, 3 cancelled, 75 paused at a gate.

A local ci run deploy deploys production for real, behind the same gates. --until and --stub are the rehearsal tools.

Environment contract (versioned with the result schema)

Variable	Contents
`CI_RUN_DIR`	The run's directory
`CI_RESULT`	Where this action writes its result
`CI_STATE`	Shared state for the run's actions
`CI_ARTIFACTS`	Attachment directory
`CI_INPUTS`	Path of a JSON file: the `inputs` of the `next` entry that routed here (`{}` if none)
`CI_ANSWER`	Path of a private file holding what the person typed at the gate, for an action with `input`. Removed when the action ends; never in events, `state.json`, arguments or unit properties
`CI_TRIGGER_RESULT`	Path of the result file that routed here (empty if none)
`CI_ROOT`, `CI_WORKFLOW`, `CI_ATTEMPT`	Main checkout; workflow name; execution number in the run
`CI_SHA`, `CI_REF`, `CI_REPO`	What's being run
`CI_RUN_ID`, `CI_ACTION`	Identity of the run and action
`CI_HOST`	`local` or `runner`. Informational; recipes must not branch on it
`CI`, `TANGLED_*`	`CI=true` and the spindle's `TANGLED_SHA`, `TANGLED_REF`, `TANGLED_REF_NAME`, `TANGLED_REF_TYPE`, `TANGLED_PIPELINE_ID`, `TANGLED_PIPELINE_KIND`, `TANGLED_REPO_*`, set in both modes so recipes can read the same names under either
per-repo `env:`	Paths exported by `repos:` entries (e.g. `SECRETS_DIR`)

Run directory

<run>/manifest.json        state.json   (cursor, history, pending gate)
      events.ndjson        (log lines, results, gates, transitions — one format everywhere)
      state/  artifacts/  results/<n>-<action>.json
      <repo>/  <extra repos>/   (worktrees; layout mirrors ~/projects)

Repos

Layout: the main checkout is <run>/<repo> and each repos: entry is <run>/<repo>/<path>, which must be a sibling (../servers → <run>/servers). Locally, without --clean, <run>/<repo> is a symlink to the working copy, so uncommitted changes are visible; with --clean it is a fresh checkout of --rev (jj revset, default @; git rev, default HEAD), i.e. exactly what the VCS tracks.
Extra repos are fetched into a bare mirror per repo ($CI_CACHE_DIR/mirrors/<repo>.git, default ~/.cache/ci) on every run, then checked out with git clone --shared at a detached commit: the forge's default branch, or the entry's rev. Checkouts borrow the mirror's objects but are independent, so a run is deleted with rm -rf and nothing is written into any repository's metadata. Each checkout's origin is the forge URL, so recipes can push (git push origin HEAD:main from the detached checkout).
repo: is a name or a full clone URL (ssh://, https:// or git@host:path). A full URL is used as written and is named by its last path element; two different URLs with the same name are an error. Its mirror is <name>-<hash of URL>.git.
A name's clone URL is a sibling of the main repo's origin (git@git.kilimanjaro.io:servers → git@git.kilimanjaro.io:data), or $CI_CLONE_URL with {repo} (e.g. git@host:{repo}).
env: on an entry exports its checkout path (e.g. SECRETS_DIR).
Every checkout's commit is recorded as a checkout event in events.ndjson; linked directories are recorded as linked.
--repo name=/path links an extra repo to a local directory instead (explicit; off by default).
ci gc [--keep 168h] deletes finished runs older than --keep (never running or paused ones; symlinked working copies are only unlinked) and compacts the mirrors. Mirrors have gc.auto=0, so objects a live checkout borrows aren't pruned under it.

Manual actions against a run: ci action --run <run> <action> (only manual: true actions, as in the appview) starts a child run that shares the parent's checkouts and $CI_STATE (and gets $CI_PARENT_RUN). It runs the action, follows only what its result explicitly routes to, and ends with the action's own status, so a rollback is recorded as its own run ("rollback done") instead of rewriting the deploy run's history. Declared gates apply.

Segments: a gate ends the process. The engine runs actions until it reaches a gate, writes a gate event, saves state.json and exits with a "paused" status. Locally it prompts inline instead (unless --detach). On the runner the daemon publishes the gate and later starts ci resume <run> --choose X in a new unit. Nothing runs while a gate waits, and runner restarts never lose a run.

Concurrency: flock on <runs>/.locks/<group>.lock (the runner passes /var/lib/ci/locks). A run takes a group's lock when it first runs an action in the group and holds it until it ends or pauses at a gate, so a run waiting for a person never blocks others. supersede: true cancels older runs of the same repo and workflow waiting at a gate in that group when a newer run reaches one. Gates past expiresAt expire on ci resume (any answer) or ci resume --expire (the runner calls it on a timer). Local runs with the appview configured (CI_APPVIEW_URL, CI_TOKEN) take their groups from the runner instead: a lease on the same file lock the runner's segments use (POST /api/locks/<group>, renewed every 30 s, released after 2 minutes without renewal), so a laptop deploy and a CI deploy can't overlap. If the runner can't be reached they warn once and lock locally; --local-locks forces that.

10. Runner daemon (CI host) #

Runs start from triggerPipeline (the appview's Run form, ci trigger), or from a sh.tangled.git.refUpdate on the bridge's /events (§5); it keeps that stream's cursor in SQLite and honours skip-ci/ci-skip push options.
For either, it fetches the repo into its bare mirror, reads .tangled/workflows/*.yml at the commit, keeps manifests with engine: devenv whose when: matches (events push and manual; branch/tag globs with * and **), and records one sh.tangled.pipeline with the raw YAML of each matching workflow. pull_request and schedule never fire.
Per run: creates the run dir and worktrees (from a persistent bare mirror per repo), then starts a segment as a transient unit:

systemd-run --unit=ci-<run>-<seg> --slice=ci.slice --uid=ci --gid=ci \
  -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes -p NoNewPrivileges=yes \
  -p "ReadWritePaths=<run> <cache>/home/<repo> <locks> /nix/var/nix/daemon-socket" \
  -p LoadCredential=age-identity:<path> -p StandardOutput=append:<run>/segment-<n>.log \
  --setenv=HOME=<cache>/home/<repo> \
  -- pasta --config-net -t none -u none -T none -U none -- ci runner-exec <run>

Run ids are <pipeline-rkey>-<workflow>; manual action runs are <run>-<action>-<tid> (child runs, DESIGN §9). The daemon prepares the run directory and state.json (engine.Create, Host runner, Pipeline AT-URI), then each segment runs ci runner-exec <run-dir> (fresh, --from <action>, or --choose N | --dismiss | --expire), always pausing at gates. A runs table (in runner.db, next to the event log) tracks status, segment, unit and the events offset already published.
Tails <run>/events.ndjson: gates and transitions are recorded as run.rtw.ci.gate / run.rtw.ci.transition; status as sh.tangled.pipeline.status (pending when prepared, running when the first segment starts, then success / failed / cancelled, or timeout for an expired gate). Waiting at a gate stays running. Manual action runs record no status for the parent's workflow.
/api/... serves runs, events, log.txt (plain text for agents), attachments, gate answers, manual actions and cancel, all behind one bearer token held by the appview; /xrpc/sh.tangled.ci.triggerPipeline starts manual pipelines.
Segments are queued behind --max-concurrent slots; a segment that can't start ends its run as failed. Gates past expiresAt are expired on a 30 s timer. With --ntfy set (it isn't today), notifications go out for gates and failed runs (optionally successes), linking to the appview's run page.
On restart, Reconcile follows runs whose unit is still active, settles finished ones, starts never-started ones, and marks runs whose process is gone without a final state as failed ("ended without finishing"). Transient units survive daemon restarts, including switch-to-configuration during a self-deploy.
Invalid workflow files are listed in the pipeline with a failed status and the validation error, so their author sees why.

Isolation (protects the host's config and services, not against attack)

Concern	Control
Writes to `/etc`, units, users, system profile	`ProtectSystem=strict`, non-root `ci` user, not in `trusted-users`. Only an explicit deploy action can change host config
Port clashes (devenv postgres vs. production postgres)	Opt-in network namespace per segment via `pasta` (egress, `pastaHostPorts` forwarded to the host). Off by default: in pasta's user namespace commands run as uid 0, root-owned files look unowned (OpenSSH refuses its config, so no SSH and no deploys) and the hostname reads `pasta-<host>`. A systemd-made namespace with pasta attached from outside would avoid that, but pasta's own sandboxing refuses to open the unit's namespace (tried; see bug 7a3d5cf/db62e70 notes)
CPU / memory / IO starvation	`ci.slice` with low `CPUWeight`/`IOWeight`, `MemoryMax`; swap or zram so builds can't OOM the forge; 1–2 concurrent runs
Disk	Run dirs expire after N days; host `nix.gc` and `min-free`/`max-free`
Nix store	Shared through the host nix-daemon socket; append-only, so growth is disk usage, not pollution

The runner daemon runs as root (it creates transient units for ci; granting ci polkit's manage-units would let a segment start root units), hands each run directory and its repo's HOME to ci before a segment starts, and is reachable on localhost only. Segments get a fixed PATH (engine, bash, coreutils, git, openssh, nix, jq, curl, just, devenv), NIX_REMOTE=daemon, CI_SELF_HOST (deploy recipes put that host last) and credentials: the age identity (SECRETS_IDENTITY) and an SSH key (for privilegedRepos only, if that's set; see SECURITY.md), found by the ci user's static ~/.ssh/config (IdentityFile ${CREDENTIALS_DIRECTORY}/ssh-key; OpenSSH reads the passwd home, and ${VAR} expands per unit), with strict host keys except the forge on localhost. nix/module.nix packages all of this as services.ci.{bridge,runner,appview}; nix/tests/e2e.nix checks it in a VM.

Multi-arch: jobs use the host nix-daemon, which routes derivations by system. x86_64 work runs natively. aarch64 work (nix build / colmena build / flake checks) goes to a builder host as a remote builder: any host with the builder role in servers' hosts.json (planned: mx99.eu.sovrn.at, the aarch64 staging host), listed in the CI host's nix.buildMachines from that data (servers/roles/ci/builders.nix). boot.binfmt.emulatedSystems = [ "aarch64-linux" ] stays as a slow fallback. The builder role (servers/roles/builder) is separate from the sovrn roles: a key-only nix-ssh login whose forced command is nix-daemon --stdio, with its own key pair (keys/nix-builder.pub, secret nix-builder-ssh-key). It is a trusted user, since the CI host's own build outputs carry no signature the builder knows; that grants nothing new, because the CI host already has root on the fleet. The builder fetches inputs from the caches itself (builders-use-substitutes), and limits builds: max-jobs = 1, cores = 1, idle CPU/IO scheduling, and a memory ceiling plus low weights on nix-daemon.service (the daemon runs every build, including those that arrive over SSH; a VM test checks the cgroup), and zram. servers' checks.x86_64-linux.builder exercises both halves. Measured on mx99 (2 vCPU / 3.8 GB): small builds are unnoticeable, but a from-source Rust compile the size of Stalwart's needs well over 4 GB. With the limits above it was throttled (MemoryHigh hit millions of times, zram full, about a third of a core of CPU) and never killed, and the host's services stayed up. Builders of this size are not what mx99 is for: such builds go to the dev machine (qemu, slow, 64 GB) or, later, a dedicated ARM builder.

Binary cache: the CI host's own /nix/store is the cache, served read-only by harmonia behind Caddy (basic auth, one shared netrc: store paths reveal host layout, even though they hold no secrets). Harmonia signs what it serves, so producers need no key: they push with nix copy --to ssh-ng://root@<ci host> (root over SSH, as deploys do); outputs a remote builder returns to the CI host's store arrive without any push:

the dev machine (64 GB RAM) builds x86_64 closures, which ci run there and the runner evaluate to identical store paths, so the runner substitutes instead of building;
aarch64 builds the runner sends to mx99 come back into the CI host's store by themselves; builds mx99 does on its own (staging previews) can be pushed too;
every other fleet host lists the cache in substituters (after cache.nixos.org), its public key in trusted-public-keys and the netrc in netrc-file, with fallback and a short connect-timeout so an outage costs time, not deploys; the ci user needs no extra trust. Implemented in servers: modules/nix-cache.nix, roles/ci/cache.nix, just cache-push, ci-cache-pin.

The cache is not a repository of project state (the runner pushes nothing to repos): losing it only costs rebuild time. GC on the CI host (min-free / max-free) must not eat it silently: pin GC roots for the current closure of each host (last N deployed generations), and watch disk (30 GB on infra.rtw.run). Only identical derivations hit; a dirty tree or different input pin builds fresh, which is correct.

Workspaces: per-run worktrees (a gate can hold a run open for hours); HOME and .cache shared per repo (accepted shared state). .devenv/ is per run; activation still hits a warm store. Host substituters include devenv.cachix.org, since the ci user can't add substituters itself.

Host sizing: infra.rtw.run stays unresized (2 vCPU / 3.8 GB, x86_64, no swap). That is enough because builds happen elsewhere: x86_64 on the dev machine, aarch64 on mx99, both pushed to the cache, so the runner mostly evaluates and substitutes (fleet eval ~1-2 GB). Requirements: zram on the host, ci.slice with MemoryMax about 2.5 GB, one concurrent run, and nix.settings.max-jobs of 1. Revisit (resize, or a dedicated ARM builder) if runs start building locally or two overlapping runs become necessary. mxb.eu.sovrn.at (4 vCPU / 8 GB) was considered as the runner host and rejected: it carries the live moods app and backup MX, and the runner holds root on the fleet.

11. Appview #

Minimal server-rendered HTML (Go templates, plain CSS, ~40 lines of vanilla JS for live logs and confirmations; every page works without JS), reachable from a phone. It keeps no state of its own: private runs come from the runner's API, and public runs are held in memory (below). Each is a Source, and one that can't be reached doesn't take the other down.

Auth: the owner signs in with a password (bcrypt hash file, e.g. mkpasswd -m bcrypt -R 12); the session is an HMAC-signed, SameSite=Strict cookie, and every form carries a session-bound CSRF token. Agents use bearer tokens from a <name> <token> file: they can read runs and plain-text logs, trigger workflows and run manual actions that aren't style: danger, never approve gates or cancel. Approvals are recorded with the owner's DID as actor. ci trigger <repo> [workflow] and ci wait <repo> [sha] use the agent API (CI_APPVIEW_URL, CI_TOKEN); ci wait exits like the run (75 when it is waiting for you).

Board: what needs the owner, then the newest run of each workflow, private and public, newest first; live logs over SSE.
Run page: timeline of actions and transitions, summaries, attachments, manual action buttons (danger rendered big and red).
Gates: prompt, the summaries of earlier actions (e.g. an nvd diff), one button per option, Dismiss. Approval requires the owner's session.
History: past runs, including reported local runs.
API: JSON status and plain-text logs for agents; ci wait uses it. Agent tokens can read, trigger and re-run, not approve gates or run danger actions.
Notifications: only the runner has any (§10), and none is configured. Public failures send nothing. If that changes, Pushover (as ~/projects/metrics uses) or email are the candidates.
Local run reporting: ci run|action|resume --report (automatic when the run executed an audited action and the appview is configured) uploads a snapshot of the run (state, events, results, artifacts; no checkouts) after each segment, so laptop deploys appear in the history. Their manual actions (Roll back) run on the runner, which checks the recorded commits out again; a run of a working copy with changes, or with --repo links, has to be acted on locally (ci action --run).
Later: ATProto OAuth login, so approvals carry the owner's DID.

Public runs #

The board shows pushes to main in the configured public repos (services.ci.appview.spindle.repos). It's for seeing what ships, so branches, pull requests and manual runs stay on Tangled. The spindle ignores its kinds filter today, so the appview checks the trigger's type and ref itself.

"Latest runs" has one row per repo and workflow across both sources, the newest, so a repo that moved to the spindle doesn't keep its last runner run there. The repo's history still shows both.

Pipelines aren't PDS records, so no firehose carries their status. The knot's com.atproto.sync.subscribeRepos says when to look: a commit touching org.tangled.git.refs, or an org.tangled.git.pushEvent. The spindle's org.tangled.ci.queryPipelines says what happened.
A repo is queried once on start, for 2 minutes after a push, every 5 s while a run is active, and on a 5-minute sweep that covers retries and anything missed. Results live in memory, so a restart just queries again.
A repo that doesn't name the spindle answers RepoNotFound, which means it has no CI, not that something is wrong.
A pipeline only has its commit's hash, so the subjects come from the knot's org.tangled.git.listCommits, at the host of the firehose URL. Each commit is asked for once and kept in memory. Without an answer a row shows just the hash.
Public runs are read-only with a link to Tangled, since retrying and cancelling need service auth. They're off the agent API, so ci wait can't hand their logs to an agent. Log text streams from org.tangled.ci.subscribePipelineLogs only while a page is open, with control characters stripped.
A pipeline doesn't list its steps, so the board's step cells come from the same log's control frames: followed while a workflow runs and read once when it finishes, up to 4 logs at a time and newest first. Only the step names and outcomes are kept, in memory, so a restart reads the cached pipelines' logs again.
If Tangled can't be reached, the board shows a notice and no "All clear". The newest result of each workflow and any active run read stale, except that last known failures stay failures. An outage can't look like a passing build.

12. Secrets #

The encrypted vault (data repo) is a repos: entry exported as SECRETS_DIR. It is cloned at its pushed revision in both modes, so only committed, pushed secrets run anywhere, and the vault sha is recorded per run.
Colmena and Justfiles call secrets decrypt <selector> exactly as today. The engine has no secrets concept and does no masking; recipes pipe secrets into tools and never print them.
The secrets tool gains (first implementation step):
- $SECRETS_DIR/.recipients: encrypt targets every listed recipient.
- secrets rekey: re-encrypt every secret to the current recipients.
- SECRETS_IDENTITY: identity file override (default ~/.config/age/identity.txt).
A deploy from the dev machine and one from CI decrypt with the same age identity, the runner's, so they share one key flow: if a deploy works by hand, it works in CI. The dev machine's own identity is a hardware key (age-plugin-yubikey), which prompts on every decrypt; it unlocks the runner's identity once per deploy instead (secrets exec, a session in $XDG_RUNTIME_DIR, removed when the deploy ends). Both are recipients of every secret; the hardware key also reads the vault directly.
Rotating the runner's identity rotates the dev machine's deploys with it: new key into the vault, swap its recipient, rekey, deploy it, and rotate what the old one could read. The hardware key never leaves the YubiKey.
The runner's identity is itself a vault secret (ci/age-identity), delivered by Colmena deployment.keys (keyCommand = [ "secrets" "decrypt" "ci/age-identity" ]) to a root-only path, and handed to segments via LoadCredential=; the engine sets SECRETS_IDENTITY to $CREDENTIALS_DIRECTORY/age-identity.
Repo access: the runner's forge deploy key (it lives on the forge host, so a local path also works).

13. CD pattern for Colmena projects (lives in project repos, not the engine) #

servers/Justfile recipes that any Colmena fleet can copy:

Recipe	Does	Result
`plan-project P`	`nix flake update P`; `colmena build` affected hosts; `nvd diff` against `readlink /run/current-system`	summary = diffs
`apply-project P`	Record each host's current system to `$CI_STATE/prev-systems.json`; `colmena apply --on <hosts>`; CI host last	—
`prod-health-check P`	Poll the project's health URLs (status code only, as sovrn's `/healthz` contract defines); N consecutive passes within a window	success, or failure offering `rollback` / "Check again", gated, with probes attached
`rollback`	Per host from `prev-systems.json`: `nix-env -p /nix/var/nix/profiles/system --set <prev> && <prev>/bin/switch-to-configuration switch`	—
`record-deploy P`	Commit and push `flake.lock` (the lock bump stays the deploy log)	—

Notes

Health URLs come from code (fleet.healthChecks). The existing monitor can later read the same list.
Rollback needs no evaluation or build, so it works with a broken flake. It does not undo state (database migrations) or uploaded keys; migrations must be backward-compatible, or recovery is roll-forward.
The CI host itself deploys last, ideally in its own gated step. If a self-deploy breaks the appview, recovery is SSH.
servers takes public projects from git+https://knot.rtw.run/rtw.run/<repo>; use --override-input locally for unpushed work.
A deploy doesn't check that the commit passed its public CI. The board shows that next to the deploy, and the person at the gate decides.

14. Local vs CI parity #

	Local default	Runner	Narrowing the gap
Engine, manifest, recipes, devenv	identical (pinned by commit)	identical	—
Source	working copy	clean worktree at sha	`--clean --rev`
Extra repos, vault	clean clones at pushed revs	same	`--repo` override
Sandbox	none, your user	`ci` user, ProtectSystem, pasta, slice	`--isolated` (`systemd-run --user` + pasta)
Nix config	your substituters, binfmt	runner's	keep both in a shared config fragment
Secrets identity	yours	runner's	both are vault recipients
Locks	local flock	runner flock	optional remote lock
History	local run dir	appview	`--report` / `audited`

Public repos get the same promise a different way: their workflows run commands through devenv shell, in Tangled's VM, so just test means the same there as on the dev machine. Those workflows aren't ci manifests.

15. Later #

Remote aarch64 builder beyond mx99; ATProto OAuth login; service-auth JWTs; monitor reading health checks; notifications for public failures. bug ls tracks the current work.