From 263223160026b8ba4b064520bfc6d6cead827bb9 Mon Sep 17 00:00:00 2001
From: Anirudh Oppiliappan <anirudh@tangled.sh>
Date: Mon, 7 Jul 2025 15:23:04 +0300
Subject: [PATCH] docs/spindle: arch, hosting and pipeline spec
Change-Id: ytzxyplmvunnqlwtprkzuulwlpqvpvpz

Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.sh>
---
 docs/spindle/architecture.md | 24 +++++++++++++++
 docs/spindle/hosting.md      | 43 ++++++++++++++++++++++++++
 docs/spindle/pipeline.md     | 59 ++++++++++++++++++++++++++++++++++++
 3 files changed, 126 insertions(+)
 create mode 100644 docs/spindle/architecture.md
 create mode 100644 docs/spindle/hosting.md
 create mode 100644 docs/spindle/pipeline.md

diff --git a/docs/spindle/architecture.md b/docs/spindle/architecture.md
new file mode 100644
index 0000000..2a91cb6
--- /dev/null
+++ b/docs/spindle/architecture.md
@@ -0,0 +1,24 @@
+# spindle architecture
+
+Spindle is a small CI runner service. Here's a high level overview of how it operates:
+
+* listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and
+[`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream.
+* when a new repo record comes through (typically when you add a spindle to a
+repo from the settings), spindle then resolves the underlying knot and
+subscribes to repo events (see:
+[`sh.tangled.pipeline`](/lexicons/pipeline.json)).
+* the spindle engine then handles execution of the pipeline, with results and
+logs beamed on the spindle event stream over wss
+
+### the engine
+
+At present, the only supported backend is Docker. Spindle executes each step in
+the pipeline in a fresh container, with state persisted across steps within the
+`/tangled/workspace` directory.
+
+The base image for the container is constructed on the fly using
+[Nixery](https://nixery.dev), which is handy for caching layers for frequently
+used packages.
+
+The pipeline manifest is [specified here](/docs/spindle/pipeline.md).
diff --git a/docs/spindle/hosting.md b/docs/spindle/hosting.md
new file mode 100644
index 0000000..51bce17
--- /dev/null
+++ b/docs/spindle/hosting.md
@@ -0,0 +1,43 @@
+# spindle self-hosting guide
+
+## prerequisites
+
+* Go
+* Docker (the only supported backend currently)
+
+## configuration
+
+Spindle is configured using environment variables. The following environment variables are available:
+
+* `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`).
+* `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`).
+* `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required).
+* `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`).
+* `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`).
+* `SPINDLE_SERVER_OWNER`: The DID of the owner (required).
+* `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`).
+* `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`).
+* `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`).
+
+## running spindle
+
+1.  **Set the environment variables.**  For example:
+
+    ```shell
+    export SPINDLE_SERVER_HOSTNAME="your-hostname"
+    export SPINDLE_SERVER_OWNER="your-did"
+    ```
+
+2.  **Build the Spindle binary.**
+
+    ```shell
+    go build -o spindle core/spindle/server.go
+    ```
+
+3.  **Run the Spindle binary.**
+
+    ```shell
+    ./spindle
+    ```
+
+Spindle will now start, connect to the Jetstream server, and begin processing pipelines.
diff --git a/docs/spindle/pipeline.md b/docs/spindle/pipeline.md
new file mode 100644
index 0000000..be15581
--- /dev/null
+++ b/docs/spindle/pipeline.md
@@ -0,0 +1,59 @@
+# spindle pipeline manifest
+
+Spindle pipelines are defined under the `.tangled/workflows` directory in a
+repo. Generally:
+
+* Pipelines are defined in YAML.
+* Dependencies can be specified from
+[Nixpkgs](https://search.nixos.org) or custom registries.
+* Environment variables can be set globally or per-step.
+
+Here's an example that uses all fields:
+
+```yaml
+# build_and_test.yaml
+when:
+  - event: ["push", "pull_request"]
+    branch: ["main", "develop"]
+  - event: ["manual"]
+
+dependencies:
+  ## from nixpkgs
+  nixpkgs:
+    - nodejs
+  ## custom registry
+  git+https://tangled.sh/@oppi.li/statix:
+    - statix
+
+steps:
+  - name: "Install dependencies"
+    command: "npm install"
+    environment:
+      NODE_ENV: "development"
+      CI: "true"
+
+  - name: "Run linter"
+    command: "npm run lint"
+
+  - name: "Run tests"
+    command: "npm test"
+    environment:
+      NODE_ENV: "test"
+      JEST_WORKERS: "2"
+
+  - name: "Build application"
+    command: "npm run build"
+    environment:
+      NODE_ENV: "production"
+
+environment:
+  BUILD_NUMBER: "123"
+  GIT_BRANCH: "main"
+
+## current repository is cloned and checked out at the target ref
+## by default.
+clone:
+  skip: false
+  depth: 50
+  submodules: true
+```
-- 
2.43.0