# Spindle configuration

> Every environment variable read by spindle and cilogs, with its default.

Spindle is configured by environment variables only. Durations use Go syntax (`5m`, `720h`) and lists are comma-separated.

## Role

| Variable             | Default      | Description                                                                                                                           |
| -------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `SPINDLE_ROLE`       | `standalone` | `standalone` ingests events and runs jobs, `mill` ingests events and places jobs on executors, `executor` dials a mill and runs jobs. |
| `SPINDLE_LOG_FORMAT` | `text`       | Format of logs on stderr: `text` or `json`.                                                                                           |

## Server

| Variable                                   | Default                                           | Description                                                                                                               |
| ------------------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `SPINDLE_SERVER_HOSTNAME`                  | required                                          | Public hostname. The spindle DID is `did:web:<hostname>`.                                                                 |
| `SPINDLE_SERVER_OWNER`                     | required in standalone and mill                   | DID of the spindle owner, always an admin.                                                                                |
| `SPINDLE_SERVER_ADMINS`                    | unset                                             | DIDs of additional admins. Each entry must be a valid DID.                                                                |
| `SPINDLE_SERVER_ADMISSION`                 | `closed`                                          | `closed` serves admins and the allow-list, `open` serves anyone who is not banned.                                        |
| `SPINDLE_SERVER_KEY_PATH`                  | required in standalone and mill                   | PEM file with the EC P-256 private key behind `/.well-known/did.json`.                                                    |
| `SPINDLE_SERVER_LISTEN_ADDR`               | `0.0.0.0:6555`                                    | HTTP listen address.                                                                                                      |
| `SPINDLE_SERVER_METRICS_LISTEN_ADDR`       | `127.0.0.1:9091`                                  | Prometheus metrics listen address.                                                                                        |
| `SPINDLE_SERVER_DB_PATH`                   | `spindle.db`                                      | SQLite database path. Also the default `--db` of the admin subcommands.                                                   |
| `SPINDLE_SERVER_REPO_DIR`                  | `repos`                                           | Directory of per-repo checkouts.                                                                                          |
| `SPINDLE_SERVER_JETSTREAM_ENDPOINT`        | `wss://jetstream1.us-west.bsky.network/subscribe` | Jetstream endpoint for live events.                                                                                       |
| `SPINDLE_SERVER_PLC_URL`                   | `https://plc.directory`                           | PLC directory for DID resolution.                                                                                         |
| `SPINDLE_SERVER_DEV`                       | `false`                                           | Development mode: allows plain HTTP and private-network addresses for knots, webhooks, clones and backfill.               |
| `SPINDLE_SERVER_DEV_EXTRA_HOSTS`           | unset                                             | Hostnames mapped to `host-gateway` in nixery workflow containers.                                                         |
| `SPINDLE_SERVER_LOG_DIR`                   | `/var/log/spindle`                                | Directory of workflow log files. In standalone, the disk artifact store when `SPINDLE_ARTIFACT_STORES_DISK_DIR` is unset. |
| `SPINDLE_SERVER_LOG_MAX_BYTES`             | `33554432`                                        | Maximum size of one workflow’s stored log, in bytes. Output past it is dropped. `0` means no cap.                         |
| `SPINDLE_SERVER_MAX_JOB_COUNT`             | `2`                                               | Pipelines that run at once.                                                                                               |
| `SPINDLE_SERVER_REPO_JOB_WORKERS`          | `256`                                             | Repos refreshed at once, shared between knots. Must be greater than zero.                                                 |
| `SPINDLE_SERVER_NOTIFY_SLOTS`              | `64`                                              | Knot notify events processed at once, shared between knots. Must be greater than zero.                                    |
| `SPINDLE_SERVER_PRIORITY_KNOTS`            | unset                                             | Knot hosts or `did:web` DIDs that get reserved repo job workers and notify slots.                                         |
| `SPINDLE_SERVER_PRIORITY_REPO_JOB_WORKERS` | `0`                                               | Repo job workers reserved for priority knots. `0` means a quarter of `SPINDLE_SERVER_REPO_JOB_WORKERS`.                   |
| `SPINDLE_SERVER_PRIORITY_NOTIFY_SLOTS`     | `0`                                               | Notify slots reserved for priority knots. `0` means a quarter of `SPINDLE_SERVER_NOTIFY_SLOTS`.                           |
| `SPINDLE_SERVER_DOCKER_SOCKET`             | unset                                             | Docker socket mounted into nixery workflow containers and set as their `DOCKER_HOST`.                                     |
| `SPINDLE_SERVER_CORS_ORIGINS`              | unset                                             | Origins allowed to call XRPC from a browser, or `*` for any. Unset disables CORS.                                         |
| `SPINDLE_SERVER_LOGS_ADDR`                 | unset                                             | `host:port` of a cilogs SSH server, returned to a knot whose push scheduled a pipeline.                                   |

## Contact and links

Served by `org.tangled.spindle.describeSpindle`.

| Variable                         | Default                         | Description                     |
| -------------------------------- | ------------------------------- | ------------------------------- |
| `SPINDLE_CONTACT_EMAIL`          | required in standalone and mill | Operator contact email.         |
| `SPINDLE_LINKS_PRIVACY_POLICY`   | unset                           | Privacy policy URL. Absolute.   |
| `SPINDLE_LINKS_TERMS_OF_SERVICE` | unset                           | Terms of service URL. Absolute. |

## Backfill

| Variable                     | Default   | Description                                                                        |
| ---------------------------- | --------- | ---------------------------------------------------------------------------------- |
| `SPINDLE_SERVER_HYDRANT_DIR` | `hydrant` | Directory of the embedded hydrant database, its API socket and the spindle cursor. |

## Secrets

| Variable                                    | Default                 | Description                                                                       |
| ------------------------------------------- | ----------------------- | --------------------------------------------------------------------------------- |
| `SPINDLE_SERVER_SECRETS_PROVIDER`           | `sqlite`                | `sqlite` stores secrets in the spindle database, `openbao` uses an OpenBao Proxy. |
| `SPINDLE_SERVER_SECRETS_OPENBAO_PROXY_ADDR` | `http://127.0.0.1:8200` | OpenBao Proxy address.                                                            |
| `SPINDLE_SERVER_SECRETS_OPENBAO_MOUNT`      | `spindle`               | KV v2 mount.                                                                      |

## Nixery engine

| Variable                                            | Default             | Description                                  |
| --------------------------------------------------- | ------------------- | -------------------------------------------- |
| `SPINDLE_NIXERY_PIPELINES_NIXERY`                   | `nixery.tangled.sh` | Nixery host that builds workflow images.     |
| `SPINDLE_NIXERY_PIPELINES_WORKFLOW_TIMEOUT`         | `5m`                | Default workflow timeout.                    |
| `SPINDLE_NIXERY_PIPELINES_MAX_JOB_MEMORY_MB`        | `6144`              | Memory limit per workflow container, in MiB. |
| `SPINDLE_NIXERY_PIPELINES_MAX_CONCURRENT_WORKFLOWS` | `8`                 | Workflow containers running at once.         |

## MicroVM engine

| Variable                                            | Default         | Description                                                                   |
| --------------------------------------------------- | --------------- | ----------------------------------------------------------------------------- |
| `SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`               | unset           | Directory of microVM images. Required by microVM workflows.                   |
| `SPINDLE_MICROVM_PIPELINES_OVERLAY_DIR`             | system temp dir | Directory of per-workflow temporary disks.                                    |
| `SPINDLE_MICROVM_PIPELINES_DEFAULT_IMAGE`           | `nixos-x86_64`  | Image used when a workflow sets no `image`.                                   |
| `SPINDLE_MICROVM_PIPELINES_AGENT_PORT`              | `10240`         | vsock port of the guest agent.                                                |
| `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM`              | `true`          | Use KVM acceleration.                                                         |
| `SPINDLE_MICROVM_PIPELINES_WORKFLOW_TIMEOUT`        | `5m`            | Default workflow timeout.                                                     |
| `SPINDLE_MICROVM_PIPELINES_AGING_THRESHOLD`         | `30s`           | Queue time after which smaller requests can no longer go ahead of a workflow. |
| `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_MEMORY_MIB`    | `0`             | Memory across all running microVMs, in MiB. `0` means unlimited.              |
| `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_VCPUS`         | `0`             | vCPUs across all running microVMs. `0` means unlimited.                       |
| `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_DISK_MIB`      | `0`             | Disk across all running microVMs, in MiB. `0` means unlimited.                |
| `SPINDLE_MICROVM_PIPELINES_MAX_WORKFLOW_MEMORY_MIB` | `0`             | Memory one workflow may request, in MiB. `0` means unlimited.                 |
| `SPINDLE_MICROVM_PIPELINES_MAX_WORKFLOW_VCPUS`      | `0`             | vCPUs one workflow may request. `0` means unlimited.                          |
| `SPINDLE_MICROVM_PIPELINES_MAX_WORKFLOW_DISK_MIB`   | `0`             | Disk one workflow may request, in MiB. `0` means unlimited.                   |

### Cgroups

| Variable                                                     | Default | Description                                                                                             |
| ------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------- |
| `SPINDLE_MICROVM_PIPELINES_ENABLE_CGROUPS`                   | `false` | Run each workflow’s processes in a per-workflow cgroup.                                                 |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_PARENT`                    | `self`  | Parent cgroup. `self` is the spindle process’s own cgroup.                                              |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_PIDS_MAX`                  | `4096`  | Process limit per workflow cgroup.                                                                      |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_SWAP_MAX_MIB`              | unset   | Swap limit per workflow cgroup, in MiB.                                                                 |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_CPU_MAX_PERCENT`           | `0`     | CPU quota as a percentage of one core. `0` caps each VM at its vCPU count, negative disables the limit. |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_IO_WEIGHT`                 | `0`     | IO weight, 1 to 10000. `0` leaves IO unlimited.                                                         |
| `SPINDLE_MICROVM_PIPELINES_CGROUP_SUPERVISOR_MEMORY_MIN_MIB` | `512`   | `memory.min` of the spindle supervisor cgroup, in MiB.                                                  |

### Debug SSH

| Variable                                            | Default        | Description                                                 |
| --------------------------------------------------- | -------------- | ----------------------------------------------------------- |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_ENABLED`       | `false`        | Allow SSH into the VM of a failed workflow.                 |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_LISTEN_ADDR`   | `0.0.0.0:2222` | Debug SSH listen address.                                   |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_HOST`          | unset          | Host in the printed SSH command.                            |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_JUMP_HOST`     | unset          | Jump host in the printed SSH command.                       |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_HOST_KEY_PATH` | unset          | Host private key. Unset generates one next to the database. |
| `SPINDLE_MICROVM_PIPELINES_DEBUG_SSH_GRACE_PERIOD`  | `5m`           | Time a failed workflow’s VM stays up after failure.         |

## Nix cache

| Variable                                   | Default | Description                                                 |
| ------------------------------------------ | ------- | ----------------------------------------------------------- |
| `SPINDLE_NIX_CACHE_READ_URLS`              | unset   | Binary caches microVM guests read from.                     |
| `SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS`    | unset   | Trusted public keys for those caches.                       |
| `SPINDLE_NIX_CACHE_UPLOAD_URL`             | unset   | Binary cache that paths built in the guest are uploaded to. |
| `SPINDLE_NIX_CACHE_REQUIRE_SIGNED_UPLOADS` | `false` | Require uploaded narinfos to be signed by a trusted key.    |
| `SPINDLE_NIX_CACHE_MAX_STAGED_MIB`         | `5120`  | Upload staging limit, in MiB. Must be greater than zero.    |
| `SPINDLE_NIX_CACHE_MAX_STAGED_OBJECTS`     | `1024`  | Upload staging object limit. Must be greater than zero.     |

## CI cache

| Variable                              | Default                      | Description                                                                                                                                 |
| ------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `SPINDLE_CACHE_BACKEND`               | unset                        | `disk` or `s3`. Unset disables the cache.                                                                                                   |
| `SPINDLE_CACHE_DISK_DIR`              | `cache` next to the database | Directory of the `disk` backend.                                                                                                            |
| `SPINDLE_CACHE_S3_BUCKET`             | unset                        | Unversioned bucket of the `s3` backend.                                                                                                     |
| `SPINDLE_CACHE_S3_PREFIX`             | `spindle/cache`              | Key prefix in the bucket.                                                                                                                   |
| `SPINDLE_CACHE_STORE_ID`              | unset                        | Identity of a shared cache store. A mill places cached workflows only on executors with the same value; unset on a mill disables the cache. |
| `SPINDLE_CACHE_RETENTION`             | `720h`                       | Time since last use after which an entry is deleted. `0` keeps entries forever.                                                             |
| `SPINDLE_CACHE_PRUNE_INTERVAL`        | `1h`                         | Interval between prunes of expired entries.                                                                                                 |
| `SPINDLE_CACHE_MAX_BYTES_PER_OWNER`   | `10737418240`                | Bytes stored across all repos of one owner. `0` means unlimited.                                                                            |
| `SPINDLE_CACHE_MAX_ENTRIES_PER_OWNER` | `1000`                       | Entries across all repos of one owner. `0` means unlimited.                                                                                 |

## Artifact stores

| Variable                            | Default     | Description                                              |
| ----------------------------------- | ----------- | -------------------------------------------------------- |
| `SPINDLE_ARTIFACT_STORES_DISK_DIR`  | unset       | Directory of the disk store.                             |
| `SPINDLE_ARTIFACT_STORES_S3_BUCKET` | unset       | Bucket of the S3 store.                                  |
| `SPINDLE_ARTIFACT_STORES_S3_REGION` | `us-east-1` | Region of the S3 store.                                  |
| `SPINDLE_S3_LOG_BUCKET`             | unset       | Deprecated alias of `SPINDLE_ARTIFACT_STORES_S3_BUCKET`. |

## Quotas

| Variable                                       | Default | Description                                                               |
| ---------------------------------------------- | ------- | ------------------------------------------------------------------------- |
| `SPINDLE_QUOTA_{USER,REPO}_WORKFLOWS`          | `0`     | Concurrent workflows. `0` means unlimited, negative is rejected.          |
| `SPINDLE_QUOTA_{USER,REPO}_VCPUS`              | `0`     | Concurrent vCPUs. `0` means unlimited, negative is rejected.              |
| `SPINDLE_QUOTA_{USER,REPO}_MEMORY_MIB`         | `0`     | Concurrent memory, in MiB. `0` means unlimited, negative is rejected.     |
| `SPINDLE_QUOTA_{USER,REPO}_DISK_MIB`           | `0`     | Concurrent disk, in MiB. `0` means unlimited, negative is rejected.       |
| `SPINDLE_QUOTA_{USER,REPO}_CACHE_STORAGE_MIB`  | `0`     | CI cache storage, in MiB. `0` means unlimited, negative is rejected.      |
| `SPINDLE_QUOTA_{USER,REPO}_WEBHOOKS`           | `0`     | Webhooks. `0` means unlimited, negative is rejected.                      |
| `SPINDLE_QUOTA_{USER,REPO}_CI_MINUTES_MONTHLY` | `0`     | CI minutes per calendar month. `0` means unlimited, negative is rejected. |

## OIDC

| Variable                | Default        | Description                                                                    |
| ----------------------- | -------------- | ------------------------------------------------------------------------------ |
| `SPINDLE_OIDC_ISSUER`   | unset          | `iss` claim of workflow ID tokens. Must be set with `SPINDLE_OIDC_KEY_PATH`.   |
| `SPINDLE_OIDC_KEY_PATH` | unset          | PEM file with an EC P-256 private key. Must be set with `SPINDLE_OIDC_ISSUER`. |
| `SPINDLE_OIDC_KEY_ID`   | key thumbprint | `kid` header. Defaults to the RFC 7638 thumbprint of the key.                  |

## Mill and executors

| Variable                               | Default | Role           | Description                                                                                                     |
| -------------------------------------- | ------- | -------------- | --------------------------------------------------------------------------------------------------------------- |
| `SPINDLE_MILL_URL`                     | unset   | executor       | Mill websocket endpoint. Required in executor, rejected in other roles.                                         |
| `SPINDLE_MILL_SHARED_SECRET`           | unset   | executor       | Executor token for the mill. Required in executor.                                                              |
| `SPINDLE_MILL_SEATS`                   | `4`     | executor       | Seats advertised to the mill.                                                                                   |
| `SPINDLE_MILL_LABELS`                  | unset   | executor       | Capability labels advertised to the mill.                                                                       |
| `SPINDLE_MILL_ARTIFACT_STORE`          | unset   | mill, executor | Name of the artifact store shared by the mill and its executors. Optional when exactly one store is configured. |
| `SPINDLE_MILL_MAX_PENDING`             | `100`   | mill           | Pending job queue limit.                                                                                        |
| `SPINDLE_MILL_RECONNECT_GRACE`         | `45s`   | mill           | Reconnect window of an executor before its leases fail.                                                         |
| `SPINDLE_MILL_CANCEL_ACK_TIMEOUT`      | `10s`   | mill           | Wait for an executor to acknowledge a cancel.                                                                   |
| `SPINDLE_MILL_CANCEL_TEARDOWN_TIMEOUT` | `10m`   | mill           | Wait for a cancelled job to tear down.                                                                          |
| `SPINDLE_MILL_DRAIN_TIMEOUT`           | `20m`   | all            | Drain time on shutdown. Must be greater than zero.                                                              |
| `SPINDLE_MILL_JUMP_LISTEN_ADDR`        | unset   | mill           | SSH jump listen address for debug sessions on executors. Requires `SPINDLE_MILL_JUMP_HOST_KEY_PATH`.            |
| `SPINDLE_MILL_JUMP_HOST_KEY_PATH`      | unset   | mill           | Host key of the jump listener.                                                                                  |
| `SPINDLE_MILL_DEBUG_EXECUTOR_PORT`     | `2223`  | mill           | Executor debug SSH port that the jump host forwards to.                                                         |
| `SPINDLE_MILL_MAX_JUMP_CONNECTIONS`    | `128`   | mill           | Concurrent jump connections. Must be greater than zero when the jump listener is set.                           |

## Scheduled workflows

| Variable                            | Default | Description                                                             |
| ----------------------------------- | ------- | ----------------------------------------------------------------------- |
| `SPINDLE_SCHEDULE_MINIMUM_INTERVAL` | `5m`    | Shortest schedule interval. Must be a positive whole number of minutes. |
| `SPINDLE_SCHEDULE_CONCURRENCY`      | `4`     | Scheduled runs started at once. Must be greater than zero.              |

## cilogs

| Variable               | Default                 | Description                                                     |
| ---------------------- | ----------------------- | --------------------------------------------------------------- |
| `CILOGS_LISTEN_ADDR`   | `0.0.0.0:3333`          | SSH listen address.                                             |
| `CILOGS_HOST_KEY_PATH` | unset                   | Host key path, created when missing. Unset uses `./id_ed25519`. |
| `CILOGS_PLC_URL`       | `https://plc.directory` | PLC directory for DID resolution.                               |
| `CILOGS_DEV`           | unset                   | `true` allows plain HTTP and private-network addresses.         |

## Telemetry

| Variable                       | Default   | Description                                              |
| ------------------------------ | --------- | -------------------------------------------------------- |
| `SPINDLE_TRACING_ENDPOINT`     | unset     | OTLP/HTTP endpoint for traces. Unset disables tracing.   |
| `SPINDLE_TRACING_SERVICE_NAME` | `spindle` | Service name on traces.                                  |
| `SPINDLE_TRACING_SAMPLE_RATIO` | `1`       | Sampling ratio, 0 to 1.                                  |
| `SPINDLE_TRACING_INSECURE`     | `false`   | Use plain HTTP to the collector.                         |
| `SPINDLE_LOGGING_ENDPOINT`     | unset     | OTLP/HTTP endpoint for logs. Unset keeps logs on stderr. |
| `SPINDLE_LOGGING_SERVICE_NAME` | `spindle` | Service name on logs.                                    |
| `SPINDLE_LOGGING_INSECURE`     | `false`   | Use plain HTTP to the collector.                         |