# Spindle

> Spindle roles, engines, how it learns about repos and pushes, and where it keeps state.

A spindle is Tangled’s CI runner: a Go binary, `spindle`, that watches repos, compiles `.tangled/workflows/*.yml` when something happens, and runs each workflow in a microVM or container. This page covers how it works inside. To write workflows, read [Writing workflows](/docs/spindles/writing-workflows/); to run one, read [Run a spindle](/docs/self-hosting/spindle/).

A mill ingests events and places workflows on executors; each executor boots a microVM per workflow.

## Roles

The same binary runs in one of three roles, set with `SPINDLE_ROLE`:

| Role                   | Ingests events | Runs workflows | Notes                                                           |
| ---------------------- | -------------- | -------------- | --------------------------------------------------------------- |
| `standalone` (default) | yes            | yes            | One host does everything                                        |
| `mill`                 | yes            | no             | Places jobs on executors; serves XRPC, logs and `/mill`         |
| `executor`             | no             | yes            | Dials the mill at `ws(s)://<mill>/mill` with a registered token |

In a mill fleet, the mill runs stand-in engines that mirror the real ones, so a remote run looks the same as a local one. Placement matches a workflow’s `runs_on` labels against executor `SPINDLE_MILL_LABELS`, then takes bids for free seats (`SPINDLE_MILL_SEATS`). The winning executor validates the image and architecture. Secrets cross the wire once, only to the winner. The mill and its executors must share one artifact store (`SPINDLE_MILL_ARTIFACT_STORE`) and, if the CI cache is on, one cache store with a matching `SPINDLE_CACHE_STORE_ID`.

Executor tokens are managed on the mill:

```sh
spindle mill executor token generate --ttl 24h
spindle mill executor add --token-file ./runner-1.token --label linux --label x86_64 runner-1
spindle mill executor list
spindle mill executor revoke runner-1
```

## Engines

| Engine    | Unit of isolation             | Notes                                                                                                                                                 |
| --------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvm` | One QEMU VM per workflow      | Images: `nixos`, `alpine`, `almalinux10`, or your own. NixOS images also accept `services`, `virtualisation`, `substituters` and flake `dependencies` |
| `nixery`  | One Docker container per step | Image built from Nix packages by [Nixery](https://nixery.dev); state shared through the workspace                                                     |
| `dummy`   | none                          | For tests                                                                                                                                             |

Each microVM guest runs **shuttle**, a Rust agent that talks to the host over vsock (port `SPINDLE_MICROVM_PIPELINES_AGENT_PORT`, default 10240). Shuttle executes steps and PTYs, activates the NixOS configuration, and proxies DNS and Nix cache traffic. The VM runs in its own namespaces, with outbound-only networking through slirp4netns, private address ranges blackholed and a filtering DNS proxy. Guests never hold cache credentials.

The microVM scheduler keeps a budget of memory, vCPUs and disk (`SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_*` and `MAX_WORKFLOW_*`) with aging and per-user fairness. Without `ENABLE_CGROUPS`, these limits are bookkeeping only; with it, cgroup v2 enforces them. The [engine README](https://tangled.org/tangled.org/core/blob/master/spindle/engines/microvm/README.md) goes deeper.

## How a spindle learns about repos

* **Records** arrive on a Jetstream-compatible stream (`SPINDLE_SERVER_JETSTREAM_ENDPOINT`; on a Tangled network, hydrant’s `/subscribe`). Spindle reads `org.tangled.repo.declaration`, `org.tangled.repo.manifest`, `org.tangled.graph.collaborator`, `org.tangled.track.ticket` and `org.tangled.review.patch`.
* **Backfill** comes from an embedded hydrant (`SPINDLE_SERVER_HYDRANT_DIR`). It only runs while a round is going: spindle starts it for new owners and for owners whose last round failed, and it fetches their `org.tangled.repo.declaration`, `repo.manifest` and `graph.collaborator` records.
* **Pushes** come from the knot. After a push to a repo whose manifest names this spindle, the knot calls the spindle’s `org.tangled.event.notifyEvent` with its own service auth, and retries from an outbox if the spindle is busy or down.

A repo is accepted when its `org.tangled.repo.manifest` record’s `spindle` field equals this spindle’s `did:web` (derived from `SPINDLE_SERVER_HOSTNAME`), its owner’s `org.tangled.repo.declaration` names it back, its owner is admitted (an admin, on the allow-list, or anyone when `SPINDLE_SERVER_ADMISSION=open`), and neither it nor its owner is banned. Spindle then sparse-clones `.tangled/workflows` from the knot over HTTPS. Changing the field to another spindle, or the manifest going missing, stops serving the repo but keeps its secrets and history. Only a repo its knot reports as deleted is wiped, secrets included.

Triggers are push, pull requests, manual runs (`org.tangled.ci.triggerPipeline`) and cron schedules. A pull request whose author has no push access waits for a collaborator’s approval.

## Storage

| What                                                                                                             | Where                                                                            |
| ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Repos, members, collaborators, pipelines, jobs, schedules, feed cursors, quotas, bans, webhooks, executor tokens | SQLite at `SPINDLE_SERVER_DB_PATH`                                               |
| Workflow sparse clones                                                                                           | `SPINDLE_SERVER_REPO_DIR`                                                        |
| Logs and artifacts                                                                                               | Disk (`SPINDLE_ARTIFACT_STORES_DISK_DIR`) or S3 (`SPINDLE_ARTIFACT_STORES_S3_*`) |
| CI cache archives                                                                                                | Disk or S3 (`SPINDLE_CACHE_*`); off when unset                                   |

Pipelines are not PDS records. Spindle serves them as `org.tangled.ci.defs#pipeline` objects over `org.tangled.ci.queryPipelines`, `getPipeline` and the `subscribePipelineLogs` websocket. Pipeline IDs are local to a spindle.

## Secrets

Repo secrets are added through `org.tangled.secret.addSecret` and stored by a provider chosen with `SPINDLE_SERVER_SECRETS_PROVIDER`: `sqlite` (default, inside the spindle DB) or `openbao` (KV v2 through an OpenBao Proxy). Secrets are injected as env vars and masked in logs.

**Trust** gates secrets, cache and OIDC. A pipeline is trusted only when its checkout comes from the target repo and, for a pull request, from a branch of it that needed no approval. Manual runs of a fork, fork and patch pull requests, and approved runs get none of them.

## Caches and the Nix substituter proxy

There are two separate caches:

* **CI cache** (`cache:` in a workflow): archives of paths keyed by lockfile hashes, stored in disk or S3, with a per-owner entry limit, a total limit and a pruner (`SPINDLE_CACHE_RETENTION`, default 720h).
* **Nix cache:** a read proxy fans out to `SPINDLE_NIX_CACHE_READ_URLS` plus the workflow’s `substituters`, and built paths are uploaded to `SPINDLE_NIX_CACHE_UPLOAD_URL`. Guests reach both through host proxies over vsock. Untrusted runs get no upload proxy. The upload cache is shared by every repository on the spindle, so it is only safe when every admitted owner is trusted. `SPINDLE_NIX_CACHE_REQUIRE_SIGNED_UPLOADS` accepts only uploads signed by `SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS`, which helps only if the signing key reaches just the repositories you trust, for example as a repository secret.

## Quotas, bans, webhooks and OIDC

* **Quotas** per user and per repo (`SPINDLE_QUOTA_{USER,REPO}_*`: workflows, vCPUs, memory, disk, cache storage, webhooks, monthly CI minutes, and the vCPUs, memory and disk of a single workflow; 0 = unlimited). Override at runtime with `spindle quota`. Cache storage covers a repo’s nix cache paths and its `cache:` entries together, and defaults to 5 GiB per user. Past it, the pruner evicts the least recently used of either.
* **Bans** with `spindle ban`, `unban` and `bans`. A ban wipes the repo’s state on the spindle.
* **Allow-list and invites** with `spindle allowlist add|remove|list` and `spindle invite`.
* **Webhooks** for push, `pull_request:*` and `repository:renamed` are stored and delivered by the spindle (`org.tangled.webhook.*`), with an SSRF-safe client. See [Webhooks](/docs/spindles/webhooks/).
* **OIDC:** with `SPINDLE_OIDC_ISSUER`, `KEY_PATH` and `KEY_ID` set, a workflow that declares `audience` gets an ES256 ID token (subject `repo:ref`) for cloud federation. Spindle serves discovery and JWKS.

## Endpoints

Spindle’s identity is `did:web:<SPINDLE_SERVER_HOSTNAME>`, the audience for service-auth calls. It listens on `SPINDLE_SERVER_LISTEN_ADDR` (default `0.0.0.0:6555`) for `/`, `/_health`, `/.well-known/did.json`, `/xrpc/*`, `/mill` (mill role) and OIDC discovery, and serves Prometheus metrics on `127.0.0.1:9091`.

[Spindle configuration](/docs/reference/spindle-config/) lists every variable, and [Workflow reference](/docs/spindles/workflows/) documents the workflow format.