Spindle

A spindle is Tangled’s CI runner: a Go binary, spindle, that watches repos, compiles .tangled/workflows/*.yml when something happens, and runs each workflow in a microVM or container. This page covers how it works inside. To write workflows, read Writing workflows; to run one, read Run a spindle.

A mill ingests events and places workflows on executors; each executor boots a microVM per workflow.

Roles

The same binary runs in one of three roles, set with SPINDLE_ROLE:

Role	Ingests events	Runs workflows	Notes
`standalone` (default)	yes	yes	One host does everything
`mill`	yes	no	Places jobs on executors; serves XRPC, logs and `/mill`
`executor`	no	yes	Dials the mill at `ws(s)://<mill>/mill` with a registered token

In a mill fleet, the mill runs stand-in engines that mirror the real ones, so a remote run looks the same as a local one. Placement matches a workflow’s runs_on labels against executor SPINDLE_MILL_LABELS, then takes bids for free seats (SPINDLE_MILL_SEATS). The winning executor validates the image and architecture. Secrets cross the wire once, only to the winner. The mill and its executors must share one artifact store (SPINDLE_MILL_ARTIFACT_STORE) and, if the CI cache is on, one cache store with a matching SPINDLE_CACHE_STORE_ID.

Executor tokens are managed on the mill:

spindle mill executor token generate --ttl 24h
spindle mill executor add --token-file ./runner-1.token --label linux --label x86_64 runner-1
spindle mill executor list
spindle mill executor revoke runner-1

Engines

Engine	Unit of isolation	Notes
`microvm`	One QEMU VM per workflow	Images: `nixos`, `alpine`, `almalinux10`, or your own. NixOS images also accept `services`, `virtualisation`, `substituters` and flake `dependencies`
`nixery`	One Docker container per step	Image built from Nix packages by Nixery; state shared through the workspace
`dummy`	none	For tests

Each microVM guest runs shuttle, a Rust agent that talks to the host over vsock (port SPINDLE_MICROVM_PIPELINES_AGENT_PORT, default 10240). Shuttle executes steps and PTYs, activates the NixOS configuration, and proxies DNS and Nix cache traffic. The VM runs in its own namespaces, with outbound-only networking through slirp4netns, private address ranges blackholed and a filtering DNS proxy. Guests never hold cache credentials.

The microVM scheduler keeps a budget of memory, vCPUs and disk (SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_* and MAX_WORKFLOW_*) with aging and per-user fairness. Without ENABLE_CGROUPS, these limits are bookkeeping only; with it, cgroup v2 enforces them. The engine README goes deeper.

How a spindle learns about repos

Records arrive on a Jetstream-compatible stream (SPINDLE_SERVER_JETSTREAM_ENDPOINT; on a Tangled network, hydrant’s /subscribe). Spindle reads org.tangled.repo.declaration, org.tangled.repo.manifest, org.tangled.graph.collaborator, org.tangled.track.ticket and org.tangled.review.patch.
Backfill comes from an embedded hydrant (SPINDLE_SERVER_HYDRANT_DIR). It only runs while a round is going: spindle starts it for new owners and for owners whose last round failed, and it fetches their org.tangled.repo.declaration, repo.manifest and graph.collaborator records.
Pushes come from the knot. After a push to a repo whose manifest names this spindle, the knot calls the spindle’s org.tangled.event.notifyEvent with its own service auth, and retries from an outbox if the spindle is busy or down.

A repo is accepted when its org.tangled.repo.manifest record’s spindle field equals this spindle’s did:web (derived from SPINDLE_SERVER_HOSTNAME), its owner’s org.tangled.repo.declaration names it back, its owner is admitted (an admin, on the allow-list, or anyone when SPINDLE_SERVER_ADMISSION=open), and neither it nor its owner is banned. Spindle then sparse-clones .tangled/workflows from the knot over HTTPS. Changing the field to another spindle, or the manifest going missing, stops serving the repo but keeps its secrets and history. Only a repo its knot reports as deleted is wiped, secrets included.

Triggers are push, pull requests, manual runs (org.tangled.ci.triggerPipeline) and cron schedules. A pull request whose author has no push access waits for a collaborator’s approval.

Storage

What	Where
Repos, members, collaborators, pipelines, jobs, schedules, feed cursors, quotas, bans, webhooks, executor tokens	SQLite at `SPINDLE_SERVER_DB_PATH`
Workflow sparse clones	`SPINDLE_SERVER_REPO_DIR`
Logs and artifacts	Disk (`SPINDLE_ARTIFACT_STORES_DISK_DIR`) or S3 (`SPINDLE_ARTIFACT_STORES_S3_*`)
CI cache archives	Disk or S3 (`SPINDLE_CACHE_*`); off when unset

Pipelines are not PDS records. Spindle serves them as org.tangled.ci.defs#pipeline objects over org.tangled.ci.queryPipelines, getPipeline and the subscribePipelineLogs websocket. Pipeline IDs are local to a spindle.

Secrets

Repo secrets are added through org.tangled.secret.addSecret and stored by a provider chosen with SPINDLE_SERVER_SECRETS_PROVIDER: sqlite (default, inside the spindle DB) or openbao (KV v2 through an OpenBao Proxy). Secrets are injected as env vars and masked in logs.

Trust gates secrets, cache and OIDC. A pipeline is trusted only when its checkout comes from the target repo and, for a pull request, from a branch of it that needed no approval. Manual runs of a fork, fork and patch pull requests, and approved runs get none of them.

Caches and the Nix substituter proxy

There are two separate caches:

CI cache (cache: in a workflow): archives of paths keyed by lockfile hashes, stored in disk or S3, with per-owner and total limits and a pruner (SPINDLE_CACHE_RETENTION, default 720h).
Nix cache: a read proxy fans out to SPINDLE_NIX_CACHE_READ_URLS plus the workflow’s substituters, and built paths are uploaded to SPINDLE_NIX_CACHE_UPLOAD_URL. Guests reach both through host proxies over vsock. Untrusted runs get no upload proxy. The upload cache is shared by every repository on the spindle, so it is only safe when every admitted owner is trusted. SPINDLE_NIX_CACHE_REQUIRE_SIGNED_UPLOADS accepts only uploads signed by SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS, which helps only if the signing key reaches just the repositories you trust, for example as a repository secret.

Quotas, bans, webhooks and OIDC

Quotas per user and per repo (SPINDLE_QUOTA_{USER,REPO}_*: workflows, vCPUs, memory, disk, cache storage, webhooks, monthly CI minutes, and the vCPUs, memory and disk of a single workflow; 0 = unlimited). Override at runtime with spindle quota.
Bans with spindle ban, unban and bans. A ban wipes the repo’s state on the spindle.
Allow-list and invites with spindle allowlist add|remove|list and spindle invite.
Webhooks for push, pull_request:* and repository:renamed are stored and delivered by the spindle (org.tangled.webhook.*), with an SSRF-safe client. See Webhooks.
OIDC: with SPINDLE_OIDC_ISSUER, KEY_PATH and KEY_ID set, a workflow that declares audience gets an ES256 ID token (subject repo:ref) for cloud federation. Spindle serves discovery and JWKS.

Endpoints

Spindle’s identity is did:web:<SPINDLE_SERVER_HOSTNAME>, the audience for service-auth calls. It listens on SPINDLE_SERVER_LISTEN_ADDR (default 0.0.0.0:6555) for /, /_health, /.well-known/did.json, /xrpc/*, /mill (mill role) and OIDC discovery, and serves Prometheus metrics on 127.0.0.1:9091.

Spindle configuration lists every variable, and Workflow reference documents the workflow format.