# ci: trusted private CD on Nix — Design

Status: built and running on `infra.rtw.run`. This describes the system as it
stands; [SECURITY.md](SECURITY.md) has the security model.

`ci` runs trusted, private deploys. It runs workflows in each repo's devenv
shell, gates deploys behind a decision a person can make from a phone, and shows
public CI from Tangled's hosted spindle on the same board. It isn't a
Tangled-compatible runner, and it doesn't run public repos' CI.

## 1. Goals and non-goals

Goals

- **Manage the fleet from a phone**: one board for deploys and public CI, so an
  urgent problem can be handled without a laptop.
- Gated CD: a person approves deploys; health is reported, rollback is one
  button. Deploys can be started from a remote Claude session (`/rc`) without
  the dev machine.
- **Local parity**: the exact same flow runs from the command line on the dev
  machine. No push-and-pray loop.
- **Generic**: nothing in the engine knows about sovrn, moods, Colmena or NixOS
  deploys. Any project can reuse it.
- Multi-arch (x86_64 + aarch64) from one runner, through Nix.

Non-goals (for now)

- Running untrusted code. Only the owner's code runs here; isolation exists to
  keep CI from changing the host's config or competing with its services, not
  to contain an attacker. Public repos' CI runs in Tangled's VMs.
- Being a spindle or a knot. Lexicons are shared only where that saves work.
- Multiple runners, autoscaling, job claiming.
- A polished UI.

## 2. Principles (the decisions behind this design)

1. **No programming in YAML.** The workflow file is an inventory of actions
   plus the default order. All logic (what to show, what happens next, when
   to ask a human) lives in code — Nix, shell, Justfiles — and reaches the
   engine through a typed JSON result contract.
2. **The engine is the product; the runner is a host for it.** The same `ci`
   binary runs locally and on the runner, pinned per repo through devenv.
3. **Steps are the happy path; actions are everything the workflow can do.**
   A step's result can route to any declared action, including happy-path
   steps.
4. **Gates are monotonic.** Code can add a gate, never remove one.
5. **Health is notify-only.** Transient errors are common after a deploy;
   the operator decides whether to roll back.
6. **Secrets are a repo.** An encrypted vault cloned at a known path; the
   `secrets` tool behaves identically everywhere. The engine has no secrets
   concept.
7. **Simplicity over isolation strength.** Sandboxed systemd units, not VMs,
   until something needs more.

## 3. System overview

```
 public repos                            private repos
 knot.rtw.run (Knot2, @rtw.run)          git@git.kilimanjaro.io:<repo> (gitolite)
      │  firehose: when to look               │
      ▼                                       │  fetched by the runner
 spindle.tangled.network (hosted)             ▼
      │  API: what happened             runner daemon ── sandboxed units ─► ci (engine)
      │                                       │          (ci.slice, user ci)  in devenv shell
      ▼                                       ▼
 appview ◄────────────────────────────── runner API (token)
    └─► browser / phone (owner), JSON + text API (agents: private runs only)

 dev machine:  `ci run <workflow>` — same engine, same manifest, same recipes
```

The URL says which side a repo is on: `knot.rtw.run` is public,
`git.kilimanjaro.io` over SSH is private, and there are no redirects between
them. `git.kilimanjaro.io/<project>` over HTTPS answers only Go's `go-import`
lookups, pointing at the knot. All of this runs on `infra.rtw.run`, and the
`servers` repo owns the forge's configuration.

| Part | Where | Responsibility |
|---|---|---|
| Public CI | Tangled's hosted spindle | `engine: microvm` workflows in public repos, in `devenv shell` so `just test` matches local |
| **Runner daemon** + **engine** | `infra.rtw.run` | Run private workflows, hold the fleet's keys, deploy |
| **Appview** | `infra.rtw.run` | Show both kinds of run, gates and actions |
| **Bridge** | `infra.rtw.run` | Turn pushes to Gitolite, reported by a post-receive hook, into events for the runner (§5) |

The bridge, runner daemon and appview are three roles of one Go binary.

## 4. Identity and records

- A private repo's identifier is `did:web:<forge-host>:<repo>` (e.g.
  `did:web:git.kilimanjaro.io:servers`). It's only a name inside this system;
  no DID document is served for it. Public repos have Tangled's `did:plc` DIDs,
  held in `org.tangled.repo.declaration` records on `@rtw.run`.
- The runner keeps its records in `runner.db` for its own history; nothing
  reads them from outside. They use `sh.tangled.pipeline`,
  `sh.tangled.pipeline.status` and `sh.tangled.git.refUpdate`, whose shapes
  fitted when this system set out to be compatible with Tangled, plus our own
  `run.rtw.ci.*`:
  - `run.rtw.ci.workflow` — the manifest schema (§6)
  - `run.rtw.ci.step.result` — the step result contract (§7)
  - `run.rtw.ci.gate` — `{pipeline, workflow, action, options[], state:
    waiting|approved|rejected|expired|superseded, actor, prompt}`
  - `run.rtw.ci.transition` — engine routing decisions, for the timeline
  - `run.rtw.ci.event` — one line of a run's `events.ndjson`
- Manual runs start through `sh.tangled.ci.triggerPipeline` on the runner's API.
- Public runs are read through `org.tangled.ci.*` on the spindle (§11).
- Lexicons are the single schema source. Go types are hand-written for the
  handful of records used, and a test checks each Go struct against its
  lexicon's properties so they cannot drift.
- Auth between services: a bearer token, on localhost.

## 5. Bridge (optional)

Every push to a gitolite repo reaches the runner through the bridge, except
`gitolite-admin`'s, whether or not a workflow wants it. Deploys are `manual`, so
they still start by hand; a workflow with `event: [push]` starts on a push:

- `services.ci.bridge.gitolite` installs a post-receive hook for every repo, which
  runs `ci bridge notify --spool <dir>`.
  It reads git's `<old> <new> <ref>` lines on stdin, takes the repo and user from
  gitolite's `GL_REPO` and `GL_USER` (or `--repo`), computes each update from the
  bare repo (changed files, commits per author, default ref, push options), writes
  one JSON file per push into the spool, pokes the bridge, and always exits 0: it
  never blocks or fails a push.
- The hook runs as `git`, which joins the `ci-bridge` group for the spool. It never
  gets the bridge's token, which can read `/events`. It can poke with `--server`
  and `--token-file` where that's wanted; the module's doesn't.
- The bridge server ingests the spool on start, on every poke and every 2 s, and
  deletes each file once its events are in the log; unparsable files are moved
  aside (`.bad`). A bridge that is down only delays events.
- It emits one `sh.tangled.git.refUpdate` per pushed ref on its token-protected
  `/events`, and the runner follows that from a cursor. `-o skip-ci` is honoured
  by the runner. The bridge doesn't read workflows.
- Event log (`internal/eventlog`, shared with the runner): SQLite
  `(created INTEGER PRIMARY KEY, rkey, nsid, event)`, where `created` is a
  monotonic UnixNano and the cursor.

## 6. Workflow manifest

`.tangled/workflows/<name>.yml`. Pure data, validated against
`run.rtw.ci.workflow`. No conditionals, expressions or loops.

```yaml
# servers/.tangled/workflows/deploy-sovrn.yml, abridged
when:                                   # matched when a run is started
  - event: [manual]
engine: devenv                          # the runner skips any other engine
repos:                                  # extra checkouts, pinned per run
  - { repo: data, path: ../data, env: SECRETS_DIR }   # exports the path

steps: [plan, deploy, health-check, record]   # the happy path

actions:
  plan:         { command: just plan-project sovrn }
  deploy:       { command: just apply-project sovrn,
                  gate: { prompt: "Deploy sovrn?" },
                  concurrency: { group: fleet, supersede: true },
                  audited: true }
  health-check: { command: just prod-health-check sovrn }
  record:       { command: just record-deploy sovrn }
  rollback:     { command: just rollback, label: Roll back, style: danger,
                  manual: true, audited: true,
                  gate: { prompt: "Roll sovrn back to what ran before this deploy?" },
                  concurrency: { group: fleet } }
```

Both Tangled's spindle and this runner read `.tangled/workflows`, and they
don't overlap. A spindle marks a workflow with an engine it doesn't have as
failed, so `engine: devenv` workflows belong in private repos, where no spindle
looks. The runner skips other engines, so a public repo's `engine: microvm`
workflow is never its concern.

A smaller project needs only what it uses:

```yaml
when: [{ event: [manual], branch: [main] }]
engine: devenv
steps: [test, deploy]
actions:
  test:   { command: devenv test }
  deploy: { command: just deploy, gate: { prompt: "Deploy?" } }
```

Action fields

| Field | Meaning |
|---|---|
| `command` | Shell command, run as `devenv shell bash -euo pipefail -c '<command>'` (§9) |
| `dir` | Working directory relative to the repo checkout (default `.`) |
| `gate` | `{ prompt, timeout }` — always ask a human before running (default timeout 24h) |
| `concurrency` | Named lock; `{ group, supersede }` — supersede cancels older runs waiting at a gate for the same group |
| `manual` | Show as a button on the run page at any time |
| `style` | `normal` or `danger` (rendering only) |
| `input` | `{ label }` — ask for one line of text when answering this action's gate; requires `gate`. The action reads it from `$CI_ANSWER` |
| `audited` | Local runs that execute this action report to the appview (§10) |

The set of actions is closed: results may only route to declared actions.
That makes the manifest an honest list of what the workflow can do, and it is
what the UI renders as buttons.

## 7. Step result contract

stdout/stderr are logs. The result is written to the file at `$CI_RESULT`.

```json
{
  "version": 1,
  "status": "failure",
  "summary": "mx99.eu.sovrn.at /healthz 503 for 4m (stalwart: connection refused)",
  "next": [
    { "action": "rollback", "prompt": "mx99 degraded after deploy. Roll back sovrn?", "then": "stop" },
    { "action": "health-check", "label": "Check again" }
  ],
  "gated": true,
  "attachments": [ { "label": "health probes", "path": "health.ndjson", "type": "ndjson" } ],
  "outputs": { "hosts": ["mx99.eu.sovrn.at"] }
}
```

| Field | Meaning |
|---|---|
| `status` | `success` · `warning` · `failure` |
| `summary` | Markdown shown on the run page, at gates and in notifications |
| `next` | Omitted → default routing. `"stop"` → end the run. Otherwise one option or a list |
| `next[].action` | A declared action (happy-path step or not) |
| `next[].then` | After that action succeeds: `continue` (resume the happy path) or `stop`. Default: `continue` for happy-path steps, `stop` otherwise |
| `next[].prompt`, `label`, `inputs` | Text for the person; data passed to the action as `$CI_INPUTS` |
| `gated` | Ask a human before taking `next` |
| `attachments` | Files under `$CI_ARTIFACTS`, rendered by type (`ndjson`, `text`, `markdown`) |
| `outputs` | Merged into `$CI_STATE/outputs.json` |

Fallbacks and errors

- No result file: exit 0 → `{status: success}`; non-zero → `{status: failure,
  next: "stop"}`. Plain recipes work unchanged.
- Invalid JSON, schema mismatch, an undeclared action, a path outside
  `$CI_ARTIFACTS`, or non-zero exit claiming success → treated as failure with
  the contract error shown.
- Each step runs with a private `/tmp`; anything worth keeping goes in
  `$CI_ARTIFACTS` or `$CI_STATE`.

A `ci-result` helper on `PATH` writes valid results from shell:

```sh
ci-result fail --summary "mx99 /healthz 503 for 4m" \
  --next rollback --prompt "Roll back sovrn?" --then stop \
  --next health-check --label "Check again" \
  --attach "health probes=health.ndjson"
```

## 8. Routing semantics

The run keeps a **cursor** into `steps`.

```
run action → read result
  ├─ next omitted  → success: happy-path action → next step; other action → stop
  │                   failure: stop
  ├─ next = "stop" → end run
  └─ next = [A, …] → gate if (any option's action declares gate) OR result.gated OR >1 option
                      gate offers each option + Dismiss (end run with this step's status)
                      no gate → run A
  naming a happy-path step moves the cursor there (forward skips, backward repeats)
  after A succeeds: then=continue → next step after cursor; then=stop → end run
```

Rules

1. **Gates are monotonic**: a declared `gate` can't be bypassed by a result.
   A choice between several options always needs a person.
2. Off-path actions end the run unless their `then` or their own result says
   otherwise. Nothing slides back onto the happy path by accident.
3. The chosen action receives `$CI_TRIGGER_RESULT` (the result that routed to
   it), `$CI_INPUTS`, and the shared `$CI_STATE`.
4. Loop guard: at most 50 transitions per run (configurable).
5. Final status, by how the run ends:
   - it runs off the end of the happy path → `success`;
   - a result says `next: "stop"` → `success`, or `failure` if that result is
     a failure (the way for an action to end a run early on purpose);
   - an action's `then: stop` → `failure` if that action is off the happy
     path ("diverted: stopped after rollback"), even when the last step had
     already started; for a happy-path step, `success` only if it was the
     last one;
   - a failure with no `next` → `failure`; a contract or engine error →
     `failure`; the loop guard → `failure`;
   - dismissing a declared gate reached by default routing → `cancelled`;
     dismissing a gate raised by a result → that result's status (`failure`
     stays `failure`, `success`/`warning` → `success`); an expired gate,
     `--until` or Ctrl-C → `cancelled`.
   These are recorded as `sh.tangled.pipeline.status` (`cancelled` and `failed`).
6. If the runner has a notifier configured (`--ntfy`; none is set today), gates
   and runs that end unsuccessfully notify (runner config, not code).
7. Approving a gate or running a `danger` action requires the owner's logged-in
   session. The agent API token (used from `/rc`) can read, trigger and re-run,
   but not approve.

## 9. Engine (`ci`)

One Go binary. Repos pin it as a devenv input (`packages = [ inputs.ci… ]`), so
the dev machine and the runner use the identical engine for a given commit.

Commands

```sh
ci validate                         # manifest + actions resolvable; schemas
ci run <wf>                         # from the working copy; gates prompt in the terminal
ci run <wf> --clean [--rev @-]      # fresh worktree of a commit, as the runner does
ci run <wf> --until plan            # stop before an action: safe rehearsal
ci run <wf> --stub health-check=fixtures/degraded.json   # fake a result, test routing
ci run <wf> --detach                # exit at gates, as on the runner
ci action <name> --no-follow        # run one action, print result + what routing would do
ci resume <run> [--choose rollback|N] [--dismiss] [--expire]   # answer a paused run's gate
ci action rollback --run <run>      # manual action against an earlier run's state
ci view <run>                       # render events.ndjson
ci runs                             # list local runs ($CI_RUNS_DIR, default ~/.local/state/ci/runs)
ci trigger <repo> <wf> [--rev]      # ask the runner for a manual pipeline
ci wait <repo> <sha>                # block until done; print failing step's logs
```

Actions run as `devenv shell bash -euo pipefail -c 'cd "$CI_ACTION_DIR"; <command>'`
started in the **main checkout**, so the workflow's repo pins the toolchain
and the engine even when an action's `dir` is another repo (`../servers`).
`--shell bash` skips devenv, for debugging only. Exit codes: 0 success, 1
failure, 2 usage, 3 cancelled, 75 paused at a gate.

A local `ci run deploy` deploys production for real, behind the same gates.
`--until` and `--stub` are the rehearsal tools.

Environment contract (versioned with the result schema)

| Variable | Contents |
|---|---|
| `CI_RUN_DIR` | The run's directory |
| `CI_RESULT` | Where this action writes its result |
| `CI_STATE` | Shared state for the run's actions |
| `CI_ARTIFACTS` | Attachment directory |
| `CI_INPUTS` | Path of a JSON file: the `inputs` of the `next` entry that routed here (`{}` if none) |
| `CI_ANSWER` | Path of a private file holding what the person typed at the gate, for an action with `input`. Removed when the action ends; never in events, `state.json`, arguments or unit properties |
| `CI_TRIGGER_RESULT` | Path of the result file that routed here (empty if none) |
| `CI_ROOT`, `CI_WORKFLOW`, `CI_ATTEMPT` | Main checkout; workflow name; execution number in the run |
| `CI_SHA`, `CI_REF`, `CI_REPO` | What's being run |
| `CI_RUN_ID`, `CI_ACTION` | Identity of the run and action |
| `CI_HOST` | `local` or `runner`. Informational; recipes must not branch on it |
| `CI`, `TANGLED_*` | `CI=true` and the spindle's `TANGLED_SHA`, `TANGLED_REF`, `TANGLED_REF_NAME`, `TANGLED_REF_TYPE`, `TANGLED_PIPELINE_ID`, `TANGLED_PIPELINE_KIND`, `TANGLED_REPO_*`, set in both modes so recipes can read the same names under either |
| per-repo `env:` | Paths exported by `repos:` entries (e.g. `SECRETS_DIR`) |

Run directory

```
<run>/manifest.json        state.json   (cursor, history, pending gate)
      events.ndjson        (log lines, results, gates, transitions — one format everywhere)
      state/  artifacts/  results/<n>-<action>.json
      <repo>/  <extra repos>/   (worktrees; layout mirrors ~/projects)
```

Repos

- Layout: the main checkout is `<run>/<repo>` and each `repos:` entry is
  `<run>/<repo>/<path>`, which must be a sibling (`../servers` →
  `<run>/servers`). Locally, without `--clean`, `<run>/<repo>` is a symlink
  to the working copy, so uncommitted changes are visible; with `--clean`
  it is a fresh checkout of `--rev` (jj revset, default `@`; git rev,
  default `HEAD`), i.e. exactly what the VCS tracks.
- Extra repos are fetched into a bare mirror per repo
  (`$CI_CACHE_DIR/mirrors/<repo>.git`, default `~/.cache/ci`) on every run,
  then checked out with `git clone --shared` at a detached commit: the
  forge's default branch, or the entry's `rev`. Checkouts borrow the
  mirror's objects but are independent, so a run is deleted with `rm -rf`
  and nothing is written into any repository's metadata. Each checkout's
  `origin` is the forge URL, so recipes can push (`git push origin
  HEAD:main` from the detached checkout).
- `repo:` is a name or a full clone URL (`ssh://`, `https://` or
  `git@host:path`). A full URL is used as written and is named by its last
  path element; two different URLs with the same name are an error. Its
  mirror is `<name>-<hash of URL>.git`.
- A name's clone URL is a sibling of the main repo's `origin`
  (`git@git.kilimanjaro.io:servers` → `git@git.kilimanjaro.io:data`), or
  `$CI_CLONE_URL` with `{repo}` (e.g. `git@host:{repo}`).
- `env:` on an entry exports its checkout path (e.g. `SECRETS_DIR`).
- Every checkout's commit is recorded as a `checkout` event in
  `events.ndjson`; linked directories are recorded as `linked`.
- `--repo name=/path` links an extra repo to a local directory instead
  (explicit; off by default).
- `ci gc [--keep 168h]` deletes finished runs older than `--keep` (never
  running or paused ones; symlinked working copies are only unlinked) and
  compacts the mirrors. Mirrors have `gc.auto=0`, so objects a live
  checkout borrows aren't pruned under it.

Manual actions against a run: `ci action --run <run> <action>` (only
`manual: true` actions, as in the appview) starts a **child run** that shares
the parent's checkouts and `$CI_STATE` (and gets `$CI_PARENT_RUN`). It runs
the action, follows only what its result explicitly routes to, and ends with
the action's own status, so a rollback is recorded as its own run ("rollback
done") instead of rewriting the deploy run's history. Declared gates apply.

Segments: a gate ends the process. The engine runs actions until it reaches a
gate, writes a `gate` event, saves `state.json` and exits with a "paused"
status. Locally it prompts inline instead (unless `--detach`). On the runner
the daemon publishes the gate and later starts `ci resume <run> --choose X` in
a new unit. Nothing runs while a gate waits, and runner restarts never lose a
run.

Concurrency: `flock` on `<runs>/.locks/<group>.lock` (the runner passes
`/var/lib/ci/locks`). A run takes a group's lock when it first runs an action
in the group and holds it until it ends or pauses at a gate, so a run waiting
for a person never blocks others. `supersede: true` cancels older runs of the
same repo and workflow waiting at a gate in that group when a newer run
reaches one. Gates past `expiresAt` expire on `ci resume` (any answer) or
`ci resume --expire` (the runner calls it on a timer). Local runs with the
appview configured (`CI_APPVIEW_URL`, `CI_TOKEN`) take their groups from the
runner instead: a lease on the same file lock the runner's segments use
(`POST /api/locks/<group>`, renewed every 30 s, released after 2 minutes
without renewal), so a laptop deploy and a CI deploy can't overlap. If the
runner can't be reached they warn once and lock locally; `--local-locks`
forces that.

## 10. Runner daemon (CI host)

- Runs start from `triggerPipeline` (the appview's Run form, `ci trigger`), or
  from a `sh.tangled.git.refUpdate` on the bridge's `/events` (§5); it keeps that
  stream's cursor in SQLite and honours
  `skip-ci`/`ci-skip` push options.
- For either, it fetches the repo into its bare mirror, reads
  `.tangled/workflows/*.yml` at the commit, keeps manifests with `engine: devenv`
  whose `when:` matches (events `push` and `manual`; `branch`/`tag` globs with
  `*` and `**`), and records one `sh.tangled.pipeline` with the raw YAML of each
  matching workflow. `pull_request` and `schedule` never fire.
- Per run: creates the run dir and worktrees (from a persistent bare mirror
  per repo), then starts a segment as a transient unit:

```
systemd-run --unit=ci-<run>-<seg> --slice=ci.slice --uid=ci --gid=ci \
  -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes -p NoNewPrivileges=yes \
  -p "ReadWritePaths=<run> <cache>/home/<repo> <locks> /nix/var/nix/daemon-socket" \
  -p LoadCredential=age-identity:<path> -p StandardOutput=append:<run>/segment-<n>.log \
  --setenv=HOME=<cache>/home/<repo> \
  -- pasta --config-net -t none -u none -T none -U none -- ci runner-exec <run>
```

- Run ids are `<pipeline-rkey>-<workflow>`; manual action runs are
  `<run>-<action>-<tid>` (child runs, DESIGN §9). The daemon prepares the run
  directory and `state.json` (`engine.Create`, Host `runner`, Pipeline
  AT-URI), then each segment runs `ci runner-exec <run-dir>` (fresh,
  `--from <action>`, or `--choose N | --dismiss | --expire`), always
  pausing at gates. A `runs` table (in `runner.db`, next to the event log)
  tracks status, segment, unit and the events offset already published.
- Tails `<run>/events.ndjson`: gates and transitions are recorded as
  `run.rtw.ci.gate` / `run.rtw.ci.transition`; status as
  `sh.tangled.pipeline.status` (`pending` when prepared, `running` when the
  first segment starts, then `success` / `failed` / `cancelled`, or `timeout`
  for an expired gate). Waiting at a gate stays `running`. Manual action runs
  record no status for the parent's workflow.
- `/api/...` serves runs, events, `log.txt` (plain text for agents), attachments,
  gate answers, manual actions and cancel, all behind one bearer token held by
  the appview; `/xrpc/sh.tangled.ci.triggerPipeline` starts manual pipelines.
- Segments are queued behind `--max-concurrent` slots; a segment that can't
  start ends its run as failed. Gates past `expiresAt` are expired on a 30 s
  timer. With `--ntfy` set (it isn't today), notifications go out for gates and
  failed runs (optionally successes), linking to the appview's run page.
- On restart, `Reconcile` follows runs whose unit is still active, settles
  finished ones, starts never-started ones, and marks runs whose process is
  gone without a final state as failed ("ended without finishing").
  Transient units survive daemon restarts, including `switch-to-configuration`
  during a self-deploy.
- Invalid workflow files are listed in the pipeline with a `failed` status
  and the validation error, so their author sees why.

Isolation (protects the host's config and services, not against attack)

| Concern | Control |
|---|---|
| Writes to `/etc`, units, users, system profile | `ProtectSystem=strict`, non-root `ci` user, not in `trusted-users`. Only an explicit deploy action can change host config |
| Port clashes (devenv postgres vs. production postgres) | Opt-in network namespace per segment via `pasta` (egress, `pastaHostPorts` forwarded to the host). **Off by default**: in pasta's user namespace commands run as uid 0, root-owned files look unowned (OpenSSH refuses its config, so no SSH and no deploys) and the hostname reads `pasta-<host>`. A systemd-made namespace with pasta attached from outside would avoid that, but pasta's own sandboxing refuses to open the unit's namespace (tried; see bug 7a3d5cf/db62e70 notes) |
| CPU / memory / IO starvation | `ci.slice` with low `CPUWeight`/`IOWeight`, `MemoryMax`; swap or zram so builds can't OOM the forge; 1–2 concurrent runs |
| Disk | Run dirs expire after N days; host `nix.gc` and `min-free`/`max-free` |
| Nix store | Shared through the host nix-daemon socket; append-only, so growth is disk usage, not pollution |

The runner daemon runs as root (it creates transient units for `ci`;
granting `ci` polkit's `manage-units` would let a segment start root units),
hands each run directory and its repo's HOME to `ci` before a segment
starts, and is reachable on localhost only. Segments get a fixed PATH
(engine, bash, coreutils, git, openssh, nix, jq, curl, just, devenv), `NIX_REMOTE=daemon`,
`CI_SELF_HOST` (deploy recipes put that host last) and credentials: the age
identity (`SECRETS_IDENTITY`) and an SSH key (for `privilegedRepos` only, if
that's set; see SECURITY.md), found by the `ci` user's
static `~/.ssh/config` (`IdentityFile ${CREDENTIALS_DIRECTORY}/ssh-key`; OpenSSH reads the passwd home,
and `${VAR}` expands per unit), with strict host keys except the forge on
localhost. `nix/module.nix` packages all of this as
`services.ci.{bridge,runner,appview}`; `nix/tests/e2e.nix` checks it in a
VM.

Multi-arch: jobs use the host nix-daemon, which routes derivations by
`system`. x86_64 work runs natively. aarch64 work (`nix build` /
`colmena build` / flake `checks`) goes to a **builder host** as a remote
builder: any host with the `builder` role in `servers`' hosts.json (planned:
`mx99.eu.sovrn.at`, the aarch64 staging host), listed in the CI host's
`nix.buildMachines` from that data (`servers/roles/ci/builders.nix`).
`boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` stays as a slow fallback.
The `builder` role (`servers/roles/builder`) is separate from the sovrn roles:
a key-only `nix-ssh` login whose forced command is `nix-daemon --stdio`, with
its own key pair (`keys/nix-builder.pub`, secret `nix-builder-ssh-key`). It is
a trusted user, since the CI host's own build outputs carry no signature the
builder knows; that grants nothing new, because the CI host already has root
on the fleet. The builder fetches inputs from the caches itself
(`builders-use-substitutes`), and limits builds: `max-jobs = 1`, `cores = 1`,
idle CPU/IO scheduling, and a memory ceiling plus low weights on
`nix-daemon.service` (the daemon runs every build, including those that
arrive over SSH; a VM test checks the cgroup), and zram. `servers`'
`checks.x86_64-linux.builder` exercises both halves.
Measured on mx99 (2 vCPU / 3.8 GB): small builds are unnoticeable, but a
from-source Rust compile the size of Stalwart's needs well over 4 GB. With the
limits above it was throttled (`MemoryHigh` hit millions of times, zram full,
about a third of a core of CPU) and never killed, and the host's services
stayed up. Builders of this size are not what mx99 is for: such builds go to
the dev machine (qemu, slow, 64 GB) or, later, a dedicated ARM builder.

Binary cache: the CI host's own `/nix/store` is the cache, served read-only by
`harmonia` behind Caddy (basic auth, one shared netrc: store paths reveal host
layout, even though they hold no secrets). Harmonia signs what it serves, so
producers need no key: they push with `nix copy --to ssh-ng://root@<ci host>`
(root over SSH, as deploys do); outputs a remote builder returns to the CI
host's store arrive without any push:

- the dev machine (64 GB RAM) builds x86_64 closures, which `ci run` there and
  the runner evaluate to identical store paths, so the runner substitutes
  instead of building;
- aarch64 builds the runner sends to mx99 come back into the CI host's store by
  themselves; builds mx99 does on its own (staging previews) can be pushed too;
- every other fleet host lists the cache in `substituters` (after
  cache.nixos.org), its public key in `trusted-public-keys` and the netrc in
  `netrc-file`, with `fallback` and a short `connect-timeout` so an outage costs
  time, not deploys; the `ci` user needs no extra trust. Implemented in
  `servers`: `modules/nix-cache.nix`, `roles/ci/cache.nix`, `just cache-push`,
  `ci-cache-pin`.

The cache is not a repository of project state (the runner pushes nothing to
repos): losing it only costs rebuild time. GC on the CI host (`min-free` /
`max-free`) must not eat it silently: pin GC roots for the current closure of
each host (last N deployed generations), and watch disk (30 GB on
`infra.rtw.run`). Only identical derivations hit; a dirty tree or different
input pin builds fresh, which is correct.

Workspaces: per-run worktrees (a gate can hold a run open for hours); `HOME`
and `.cache` shared per repo (accepted shared state). `.devenv/` is per run;
activation still hits a warm store. Host substituters include
`devenv.cachix.org`, since the `ci` user can't add substituters itself.

Host sizing: `infra.rtw.run` stays **unresized** (2 vCPU / 3.8 GB, x86_64,
no swap). That is enough because builds happen elsewhere: x86_64 on the dev
machine, aarch64 on mx99, both pushed to the cache, so the runner mostly
evaluates and substitutes (fleet eval ~1-2 GB). Requirements: zram on the
host, `ci.slice` with `MemoryMax` about 2.5 GB, one concurrent run, and
`nix.settings.max-jobs` of 1. Revisit (resize, or a dedicated ARM builder) if
runs start building locally or two overlapping runs become necessary.
`mxb.eu.sovrn.at` (4 vCPU / 8 GB) was considered as the runner host and
rejected: it carries the live moods app and backup MX, and the runner holds
root on the fleet.

## 11. Appview

Minimal server-rendered HTML (Go templates, plain CSS, ~40 lines of vanilla
JS for live logs and confirmations; every page works without JS), reachable
from a phone. It keeps no state of its own: private runs come from the runner's
API, and public runs are held in memory (below). Each is a `Source`, and one that
can't be reached doesn't take the other down.

Auth: the owner signs in with a password (bcrypt hash file, e.g.
`mkpasswd -m bcrypt -R 12`); the session is an HMAC-signed, `SameSite=Strict`
cookie, and every form carries a session-bound CSRF token. Agents use bearer
tokens from a `<name> <token>` file: they can read runs and plain-text logs,
trigger workflows and run manual actions that aren't `style: danger`, never
approve gates or cancel. Approvals are recorded with the owner's DID as
actor. `ci trigger <repo> [workflow]` and `ci wait <repo> [sha]` use the agent
API (`CI_APPVIEW_URL`, `CI_TOKEN`); `ci wait` exits like the run (75 when it
is waiting for you).

- **Board**: what needs the owner, then the newest run of each workflow, private
  and public, newest first; live logs over SSE.
- **Run page**: timeline of actions and transitions, summaries, attachments,
  manual action buttons (`danger` rendered big and red).
- **Gates**: prompt, the summaries of earlier actions (e.g. an `nvd` diff), one
  button per option, Dismiss. Approval requires the owner's session.
- **History**: past runs, including reported local runs.
- **API**: JSON status and plain-text logs for agents; `ci wait` uses it.
  Agent tokens can read, trigger and re-run, not approve gates or run `danger`
  actions.
- **Notifications**: only the runner has any (§10), and none is configured.
  Public failures send nothing. If that changes, Pushover (as `~/projects/metrics`
  uses) or email are the candidates.
- **Local run reporting**: `ci run|action|resume --report` (automatic when
  the run executed an `audited` action and the appview is configured)
  uploads a snapshot of the run (state, events, results, artifacts; no
  checkouts) after each segment, so laptop deploys appear in the history.
  Their manual actions (Roll back) run on the runner, which checks the
  recorded commits out again; a run of a working copy with changes, or with
  `--repo` links, has to be acted on locally (`ci action --run`).
- Later: ATProto OAuth login, so approvals carry the owner's DID.

### Public runs

The board shows **pushes to `main`** in the configured public repos
(`services.ci.appview.spindle.repos`). It's for seeing what ships, so branches,
pull requests and manual runs stay on Tangled. The spindle ignores its `kinds`
filter today, so the appview checks the trigger's type and ref itself.

"Latest runs" has one row per repo and workflow across both sources, the newest,
so a repo that moved to the spindle doesn't keep its last runner run there. The
repo's history still shows both.

- Pipelines aren't PDS records, so no firehose carries their status. The knot's
  `com.atproto.sync.subscribeRepos` says when to look: a commit touching
  `org.tangled.git.refs`, or an `org.tangled.git.pushEvent`. The spindle's
  `org.tangled.ci.queryPipelines` says what happened.
- A repo is queried once on start, for 2 minutes after a push, every 5 s while a
  run is active, and on a 5-minute sweep that covers retries and anything missed.
  Results live in memory, so a restart just queries again.
- A repo that doesn't name the spindle answers `RepoNotFound`, which means it has
  no CI, not that something is wrong.
- A pipeline only has its commit's hash, so the subjects come from the knot's
  `org.tangled.git.listCommits`, at the host of the firehose URL. Each commit is
  asked for once and kept in memory. Without an answer a row shows just the hash.
- Public runs are read-only with a link to Tangled, since retrying and cancelling
  need service auth. They're off the agent API, so `ci wait` can't hand their logs
  to an agent. Log text streams from `org.tangled.ci.subscribePipelineLogs` only
  while a page is open, with control characters stripped.
- A pipeline doesn't list its steps, so the board's step cells come from the same
  log's control frames: followed while a workflow runs and read once when it
  finishes, up to 4 logs at a time and newest first. Only the step names and
  outcomes are kept, in memory, so a restart reads the cached pipelines' logs again.
- If Tangled can't be reached, the board shows a notice and no "All clear". The
  newest result of each workflow and any active run read `stale`, except that last
  known failures stay failures. An outage can't look like a passing build.

## 12. Secrets

- The encrypted vault (`data` repo) is a `repos:` entry exported as
  `SECRETS_DIR`. It is cloned at its pushed revision in both modes, so only
  committed, pushed secrets run anywhere, and the vault sha is recorded per
  run.
- Colmena and Justfiles call `secrets decrypt <selector>` exactly as today.
  The engine has no secrets concept and does no masking; recipes pipe secrets
  into tools and never print them.
- The `secrets` tool gains (first implementation step):
  - `$SECRETS_DIR/.recipients`: `encrypt` targets every listed recipient.
  - `secrets rekey`: re-encrypt every secret to the current recipients.
  - `SECRETS_IDENTITY`: identity file override (default
    `~/.config/age/identity.txt`).
- A deploy from the dev machine and one from CI decrypt with the **same**
  age identity, the runner's, so they share one key flow: if a deploy works
  by hand, it works in CI. The dev machine's own identity is a hardware key
  (age-plugin-yubikey), which prompts on every decrypt; it unlocks the
  runner's identity once per deploy instead (`secrets exec`, a session in
  `$XDG_RUNTIME_DIR`, removed when the deploy ends). Both are recipients of
  every secret; the hardware key also reads the vault directly.
- Rotating the runner's identity rotates the dev machine's deploys with it:
  new key into the vault, swap its recipient, rekey, deploy it, and rotate
  what the old one could read. The hardware key never leaves the YubiKey.
- The runner's identity is itself a vault secret (`ci/age-identity`),
  delivered by Colmena `deployment.keys` (`keyCommand = [ "secrets"
  "decrypt" "ci/age-identity" ]`) to a root-only path, and handed to segments
  via `LoadCredential=`; the engine sets `SECRETS_IDENTITY` to
  `$CREDENTIALS_DIRECTORY/age-identity`.
- Repo access: the runner's forge deploy key (it lives on the forge host, so a
  local path also works).

## 13. CD pattern for Colmena projects (lives in project repos, not the engine)

`servers/Justfile` recipes that any Colmena fleet can copy:

| Recipe | Does | Result |
|---|---|---|
| `plan-project P` | `nix flake update P`; `colmena build` affected hosts; `nvd diff` against `readlink /run/current-system` | summary = diffs |
| `apply-project P` | Record each host's current system to `$CI_STATE/prev-systems.json`; `colmena apply --on <hosts>`; CI host last | — |
| `prod-health-check P` | Poll the project's health URLs (status code only, as sovrn's `/healthz` contract defines); N consecutive passes within a window | success, or failure offering `rollback` / "Check again", gated, with probes attached |
| `rollback` | Per host from `prev-systems.json`: `nix-env -p /nix/var/nix/profiles/system --set <prev> && <prev>/bin/switch-to-configuration switch` | — |
| `record-deploy P` | Commit and push `flake.lock` (the lock bump stays the deploy log) | — |

Notes

- Health URLs come from code (`fleet.healthChecks`). The existing monitor can
  later read the same list.
- Rollback needs no evaluation or build, so it works with a broken flake.
  It does not undo state (database migrations) or uploaded keys; migrations
  must be backward-compatible, or recovery is roll-forward.
- The CI host itself deploys last, ideally in its own gated step. If a
  self-deploy breaks the appview, recovery is SSH.
- `servers` takes public projects from `git+https://knot.rtw.run/rtw.run/<repo>`;
  use `--override-input` locally for unpushed work.
- A deploy doesn't check that the commit passed its public CI. The board shows
  that next to the deploy, and the person at the gate decides.

## 14. Local vs CI parity

| | Local default | Runner | Narrowing the gap |
|---|---|---|---|
| Engine, manifest, recipes, devenv | identical (pinned by commit) | identical | — |
| Source | working copy | clean worktree at sha | `--clean --rev` |
| Extra repos, vault | clean clones at pushed revs | same | `--repo` override |
| Sandbox | none, your user | `ci` user, ProtectSystem, pasta, slice | `--isolated` (`systemd-run --user` + pasta) |
| Nix config | your substituters, binfmt | runner's | keep both in a shared config fragment |
| Secrets identity | yours | runner's | both are vault recipients |
| Locks | local flock | runner flock | optional remote lock |
| History | local run dir | appview | `--report` / `audited` |

Public repos get the same promise a different way: their workflows run commands
through `devenv shell`, in Tangled's VM, so `just test` means the same there as on
the dev machine. Those workflows aren't `ci` manifests.

## 15. Later

Remote aarch64 builder beyond mx99; ATProto OAuth login; service-auth JWTs;
monitor reading health checks; notifications for public failures.
`bug ls` tracks the current work.