commit 0abad048f3cbe6be5f6ef3d47360b480c0d40610 · tangled.org/core

-136

docs/contributing.md

··· 1 - # tangled contributing guide 2 - 3 - ## commit guidelines 4 - 5 - We follow a commit style similar to the Go project. Please keep commits: 6 - 7 - * **atomic**: each commit should represent one logical change 8 - * **descriptive**: the commit message should clearly describe what the 9 - change does and why it's needed 10 - 11 - ### message format 12 - 13 - ``` 14 - <service/top-level directory>/<affected package/directory>: <short summary of change> 15 - 16 - 17 - Optional longer description can go here, if necessary. Explain what the 18 - change does and why, especially if not obvious. Reference relevant 19 - issues or PRs when applicable. These can be links for now since we don't 20 - auto-link issues/PRs yet. 21 - ``` 22 - 23 - Here are some examples: 24 - 25 - ``` 26 - appview/state: fix token expiry check in middleware 27 - 28 - The previous check did not account for clock drift, leading to premature 29 - token invalidation. 30 - ``` 31 - 32 - ``` 33 - knotserver/git/service: improve error checking in upload-pack 34 - ``` 35 - 36 - 37 - ### general notes 38 - 39 - - PRs get merged "as-is" (fast-forward) -- like applying a patch-series 40 - using `git am`. At present, there is no squashing -- so please author 41 - your commits as they would appear on `master`, following the above 42 - guidelines. 43 - - If there is a lot of nesting, for example "appview: 44 - pages/templates/repo/fragments: ...", these can be truncated down to 45 - just "appview: repo/fragments: ...". If the change affects a lot of 46 - subdirectories, you may abbreviate to just the top-level names, e.g. 47 - "appview: ..." or "knotserver: ...". 48 - - Keep commits lowercased with no trailing period. 49 - - Use the imperative mood in the summary line (e.g., "fix bug" not 50 - "fixed bug" or "fixes bug"). 51 - - Try to keep the summary line under 72 characters, but we aren't too 52 - fussed about this. 53 - - Follow the same formatting for PR titles if filled manually. 54 - - Don't include unrelated changes in the same commit. 55 - - Avoid noisy commit messages like "wip" or "final fix"—rewrite history 56 - before submitting if necessary. 57 - 58 - ## code formatting 59 - 60 - We use a variety of tools to format our code, and multiplex them with 61 - [`treefmt`](https://treefmt.com): all you need to do to format your changes 62 - is run `nix run .#fmt` (or just `treefmt` if you're in the devshell). 63 - 64 - ## proposals for bigger changes 65 - 66 - Small fixes like typos, minor bugs, or trivial refactors can be 67 - submitted directly as PRs. 68 - 69 - For larger changes—especially those introducing new features, significant 70 - refactoring, or altering system behavior—please open a proposal first. This 71 - helps us evaluate the scope, design, and potential impact before implementation. 72 - 73 - ### proposal format 74 - 75 - Create a new issue titled: 76 - 77 - ``` 78 - proposal: <affected scope>: <summary of change> 79 - ``` 80 - 81 - In the description, explain: 82 - 83 - - What the change is 84 - - Why it's needed 85 - - How you plan to implement it (roughly) 86 - - Any open questions or tradeoffs 87 - 88 - We'll use the issue thread to discuss and refine the idea before moving 89 - forward. 90 - 91 - ## developer certificate of origin (DCO) 92 - 93 - We require all contributors to certify that they have the right to 94 - submit the code they're contributing. To do this, we follow the 95 - [Developer Certificate of Origin 96 - (DCO)](https://developercertificate.org/). 97 - 98 - By signing your commits, you're stating that the contribution is your 99 - own work, or that you have the right to submit it under the project's 100 - license. This helps us keep things clean and legally sound. 101 - 102 - To sign your commit, just add the `-s` flag when committing: 103 - 104 - ```sh 105 - git commit -s -m "your commit message" 106 - ``` 107 - 108 - This appends a line like: 109 - 110 - ``` 111 - Signed-off-by: Your Name <your.email@example.com> 112 - ``` 113 - 114 - We won't merge commits if they aren't signed off. If you forget, you can 115 - amend the last commit like this: 116 - 117 - ```sh 118 - git commit --amend -s 119 - ``` 120 - 121 - If you're submitting a PR with multiple commits, make sure each one is 122 - signed. 123 - 124 - For [jj](https://jj-vcs.github.io/jj/latest/) users, you can run the following command 125 - to make it sign off commits in the tangled repo: 126 - 127 - ```shell 128 - # Safety check, should say "No matching config key..." 129 - jj config list templates.commit_trailers 130 - # The command below may need to be adjusted if the command above returned something. 131 - jj config set --repo templates.commit_trailers "format_signed_off_by_trailer(self)" 132 - ``` 133 - 134 - Refer to the [jj 135 - documentation](https://jj-vcs.github.io/jj/latest/config/#commit-trailers) 136 - for more information.

-172

docs/hacking.md

··· 1 - # hacking on tangled 2 - 3 - We highly recommend [installing 4 - nix](https://nixos.org/download/) (the package manager) 5 - before working on the codebase. The nix flake provides a lot 6 - of helpers to get started and most importantly, builds and 7 - dev shells are entirely deterministic. 8 - 9 - To set up your dev environment: 10 - 11 - ```bash 12 - nix develop 13 - ``` 14 - 15 - Non-nix users can look at the `devShell` attribute in the 16 - `flake.nix` file to determine necessary dependencies. 17 - 18 - ## running the appview 19 - 20 - The nix flake also exposes a few `app` attributes (run `nix 21 - flake show` to see a full list of what the flake provides), 22 - one of the apps runs the appview with the `air` 23 - live-reloader: 24 - 25 - ```bash 26 - TANGLED_DEV=true nix run .#watch-appview 27 - 28 - # TANGLED_DB_PATH might be of interest to point to 29 - # different sqlite DBs 30 - 31 - # in a separate shell, you can live-reload tailwind 32 - nix run .#watch-tailwind 33 - ``` 34 - 35 - To authenticate with the appview, you will need redis and 36 - OAUTH JWKs to be setup: 37 - 38 - ``` 39 - # oauth jwks should already be setup by the nix devshell: 40 - echo $TANGLED_OAUTH_CLIENT_SECRET 41 - z42ty4RT1ovnTopY8B8ekz9NuziF2CuMkZ7rbRFpAR9jBqMc 42 - 43 - echo $TANGLED_OAUTH_CLIENT_KID 44 - 1761667908 45 - 46 - # if not, you can set it up yourself: 47 - goat key generate -t P-256 48 - Key Type: P-256 / secp256r1 / ES256 private key 49 - Secret Key (Multibase Syntax): save this securely (eg, add to password manager) 50 - z42tuPDKRfM2mz2Kv953ARen2jmrPA8S9LX9tRq4RVcUMwwL 51 - Public Key (DID Key Syntax): share or publish this (eg, in DID document) 52 - did:key:zDnaeUBxtG6Xuv3ATJE4GaWeyXM3jyamJsZw3bSPpxx4bNXDR 53 - 54 - # the secret key from above 55 - export TANGLED_OAUTH_CLIENT_SECRET="z42tuP..." 56 - 57 - # run redis in at a new shell to store oauth sessions 58 - redis-server 59 - ``` 60 - 61 - ## running knots and spindles 62 - 63 - An end-to-end knot setup requires setting up a machine with 64 - `sshd`, `AuthorizedKeysCommand`, and git user, which is 65 - quite cumbersome. So the nix flake provides a 66 - `nixosConfiguration` to do so. 67 - 68 - <details> 69 - <summary><strong>MacOS users will have to setup a Nix Builder first</strong></summary> 70 - 71 - In order to build Tangled's dev VM on macOS, you will 72 - first need to set up a Linux Nix builder. The recommended 73 - way to do so is to run a [`darwin.linux-builder` 74 - VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder) 75 - and to register it in `nix.conf` as a builder for Linux 76 - with the same architecture as your Mac (`linux-aarch64` if 77 - you are using Apple Silicon). 78 - 79 - > IMPORTANT: You must build `darwin.linux-builder` somewhere other than inside 80 - > the tangled repo so that it doesn't conflict with the other VM. For example, 81 - > you can do 82 - > 83 - > ```shell 84 - > cd $(mktemp -d buildervm.XXXXX) && nix run nixpkgs#darwin.linux-builder 85 - > ``` 86 - > 87 - > to store the builder VM in a temporary dir. 88 - > 89 - > You should read and follow [all the other intructions][darwin builder vm] to 90 - > avoid subtle problems. 91 - 92 - Alternatively, you can use any other method to set up a 93 - Linux machine with `nix` installed that you can `sudo ssh` 94 - into (in other words, root user on your Mac has to be able 95 - to ssh into the Linux machine without entering a password) 96 - and that has the same architecture as your Mac. See 97 - [remote builder 98 - instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements) 99 - for how to register such a builder in `nix.conf`. 100 - 101 - > WARNING: If you'd like to use 102 - > [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or 103 - > [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo 104 - > ssh` works can be tricky. It seems to be [possible with 105 - > Orbstack](https://github.com/orgs/orbstack/discussions/1669). 106 - 107 - </details> 108 - 109 - To begin, grab your DID from http://localhost:3000/settings. 110 - Then, set `TANGLED_VM_KNOT_OWNER` and 111 - `TANGLED_VM_SPINDLE_OWNER` to your DID. You can now start a 112 - lightweight NixOS VM like so: 113 - 114 - ```bash 115 - nix run --impure .#vm 116 - 117 - # type `poweroff` at the shell to exit the VM 118 - ``` 119 - 120 - This starts a knot on port 6444, a spindle on port 6555 121 - with `ssh` exposed on port 2222. 122 - 123 - Once the services are running, head to 124 - http://localhost:3000/settings/knots and hit verify. It should 125 - verify the ownership of the services instantly if everything 126 - went smoothly. 127 - 128 - You can push repositories to this VM with this ssh config 129 - block on your main machine: 130 - 131 - ```bash 132 - Host nixos-shell 133 - Hostname localhost 134 - Port 2222 135 - User git 136 - IdentityFile ~/.ssh/my_tangled_key 137 - ``` 138 - 139 - Set up a remote called `local-dev` on a git repo: 140 - 141 - ```bash 142 - git remote add local-dev git@nixos-shell:user/repo 143 - git push local-dev main 144 - ``` 145 - 146 - ### running a spindle 147 - 148 - The above VM should already be running a spindle on 149 - `localhost:6555`. Head to http://localhost:3000/settings/spindles and 150 - hit verify. You can then configure each repository to use 151 - this spindle and run CI jobs. 152 - 153 - Of interest when debugging spindles: 154 - 155 - ``` 156 - # service logs from journald: 157 - journalctl -xeu spindle 158 - 159 - # CI job logs from disk: 160 - ls /var/log/spindle 161 - 162 - # debugging spindle db: 163 - sqlite3 /var/lib/spindle/spindle.db 164 - 165 - # litecli has a nicer REPL interface: 166 - litecli /var/lib/spindle/spindle.db 167 - ``` 168 - 169 - If for any reason you wish to disable either one of the 170 - services in the VM, modify [nix/vm.nix](/nix/vm.nix) and set 171 - `services.tangled.spindle.enable` (or 172 - `services.tangled.knot.enable`) to `false`.

-214

docs/knot-hosting.md

··· 1 - # knot self-hosting guide 2 - 3 - So you want to run your own knot server? Great! Here are a few prerequisites: 4 - 5 - 1. A server of some kind (a VPS, a Raspberry Pi, etc.). Preferably running a Linux distribution of some kind. 6 - 2. A (sub)domain name. People generally use `knot.example.com`. 7 - 3. A valid SSL certificate for your domain. 8 - 9 - There's a couple of ways to get started: 10 - * NixOS: refer to 11 - [flake.nix](https://tangled.sh/@tangled.sh/core/blob/master/flake.nix) 12 - * Docker: Documented at 13 - [@tangled.sh/knot-docker](https://tangled.sh/@tangled.sh/knot-docker) 14 - (community maintained: support is not guaranteed!) 15 - * Manual: Documented below. 16 - 17 - ## manual setup 18 - 19 - First, clone this repository: 20 - 21 - ``` 22 - git clone https://tangled.org/@tangled.org/core 23 - ``` 24 - 25 - Then, build the `knot` CLI. This is the knot administration and operation tool. 26 - For the purpose of this guide, we're only concerned with these subcommands: 27 - 28 - * `knot server`: the main knot server process, typically run as a 29 - supervised service 30 - * `knot guard`: handles role-based access control for git over SSH 31 - (you'll never have to run this yourself) 32 - * `knot keys`: fetches SSH keys associated with your knot; we'll use 33 - this to generate the SSH `AuthorizedKeysCommand` 34 - 35 - ``` 36 - cd core 37 - export CGO_ENABLED=1 38 - go build -o knot ./cmd/knot 39 - ``` 40 - 41 - Next, move the `knot` binary to a location owned by `root` -- 42 - `/usr/local/bin/` is a good choice. Make sure the binary itself is also owned by `root`: 43 - 44 - ``` 45 - sudo mv knot /usr/local/bin/knot 46 - sudo chown root:root /usr/local/bin/knot 47 - ``` 48 - 49 - This is necessary because SSH `AuthorizedKeysCommand` requires [really 50 - specific permissions](https://stackoverflow.com/a/27638306). The 51 - `AuthorizedKeysCommand` specifies a command that is run by `sshd` to 52 - retrieve a user's public SSH keys dynamically for authentication. Let's 53 - set that up. 54 - 55 - ``` 56 - sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF 57 - Match User git 58 - AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys 59 - AuthorizedKeysCommandUser nobody 60 - EOF 61 - ``` 62 - 63 - Then, reload `sshd`: 64 - 65 - ``` 66 - sudo systemctl reload ssh 67 - ``` 68 - 69 - Next, create the `git` user. We'll use the `git` user's home directory 70 - to store repositories: 71 - 72 - ``` 73 - sudo adduser git 74 - ``` 75 - 76 - Create `/home/git/.knot.env` with the following, updating the values as 77 - necessary. The `KNOT_SERVER_OWNER` should be set to your 78 - DID, you can find your DID in the [Settings](https://tangled.sh/settings) page. 79 - 80 - ``` 81 - KNOT_REPO_SCAN_PATH=/home/git 82 - KNOT_SERVER_HOSTNAME=knot.example.com 83 - APPVIEW_ENDPOINT=https://tangled.sh 84 - KNOT_SERVER_OWNER=did:plc:foobar 85 - KNOT_SERVER_INTERNAL_LISTEN_ADDR=127.0.0.1:5444 86 - KNOT_SERVER_LISTEN_ADDR=127.0.0.1:5555 87 - ``` 88 - 89 - If you run a Linux distribution that uses systemd, you can use the provided 90 - service file to run the server. Copy 91 - [`knotserver.service`](/systemd/knotserver.service) 92 - to `/etc/systemd/system/`. Then, run: 93 - 94 - ``` 95 - systemctl enable knotserver 96 - systemctl start knotserver 97 - ``` 98 - 99 - The last step is to configure a reverse proxy like Nginx or Caddy to front your 100 - knot. Here's an example configuration for Nginx: 101 - 102 - ``` 103 - server { 104 - listen 80; 105 - listen [::]:80; 106 - server_name knot.example.com; 107 - 108 - location / { 109 - proxy_pass http://localhost:5555; 110 - proxy_set_header Host $host; 111 - proxy_set_header X-Real-IP $remote_addr; 112 - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 113 - proxy_set_header X-Forwarded-Proto $scheme; 114 - } 115 - 116 - # wss endpoint for git events 117 - location /events { 118 - proxy_set_header X-Forwarded-For $remote_addr; 119 - proxy_set_header Host $http_host; 120 - proxy_set_header Upgrade websocket; 121 - proxy_set_header Connection Upgrade; 122 - proxy_pass http://localhost:5555; 123 - } 124 - # additional config for SSL/TLS go here. 125 - } 126 - 127 - ``` 128 - 129 - Remember to use Let's Encrypt or similar to procure a certificate for your 130 - knot domain. 131 - 132 - You should now have a running knot server! You can finalize 133 - your registration by hitting the `verify` button on the 134 - [/settings/knots](https://tangled.org/settings/knots) page. This simply creates 135 - a record on your PDS to announce the existence of the knot. 136 - 137 - ### custom paths 138 - 139 - (This section applies to manual setup only. Docker users should edit the mounts 140 - in `docker-compose.yml` instead.) 141 - 142 - Right now, the database and repositories of your knot lives in `/home/git`. You 143 - can move these paths if you'd like to store them in another folder. Be careful 144 - when adjusting these paths: 145 - 146 - * Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent 147 - any possible side effects. Remember to restart it once you're done. 148 - * Make backups before moving in case something goes wrong. 149 - * Make sure the `git` user can read and write from the new paths. 150 - 151 - #### database 152 - 153 - As an example, let's say the current database is at `/home/git/knotserver.db`, 154 - and we want to move it to `/home/git/database/knotserver.db`. 155 - 156 - Copy the current database to the new location. Make sure to copy the `.db-shm` 157 - and `.db-wal` files if they exist. 158 - 159 - ``` 160 - mkdir /home/git/database 161 - cp /home/git/knotserver.db* /home/git/database 162 - ``` 163 - 164 - In the environment (e.g. `/home/git/.knot.env`), set `KNOT_SERVER_DB_PATH` to 165 - the new file path (_not_ the directory): 166 - 167 - ``` 168 - KNOT_SERVER_DB_PATH=/home/git/database/knotserver.db 169 - ``` 170 - 171 - #### repositories 172 - 173 - As an example, let's say the repositories are currently in `/home/git`, and we 174 - want to move them into `/home/git/repositories`. 175 - 176 - Create the new folder, then move the existing repositories (if there are any): 177 - 178 - ``` 179 - mkdir /home/git/repositories 180 - # move all DIDs into the new folder; these will vary for you! 181 - mv /home/git/did:plc:wshs7t2adsemcrrd4snkeqli /home/git/repositories 182 - ``` 183 - 184 - In the environment (e.g. `/home/git/.knot.env`), update `KNOT_REPO_SCAN_PATH` 185 - to the new directory: 186 - 187 - ``` 188 - KNOT_REPO_SCAN_PATH=/home/git/repositories 189 - ``` 190 - 191 - Similarly, update your `sshd` `AuthorizedKeysCommand` to use the updated 192 - repository path: 193 - 194 - ``` 195 - sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF 196 - Match User git 197 - AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys -git-dir /home/git/repositories 198 - AuthorizedKeysCommandUser nobody 199 - EOF 200 - ``` 201 - 202 - Make sure to restart your SSH server! 203 - 204 - #### MOTD (message of the day) 205 - 206 - To configure the MOTD used ("Welcome to this knot!" by default), edit the 207 - `/home/git/motd` file: 208 - 209 - ``` 210 - printf "Hi from this knot!\n" > /home/git/motd 211 - ``` 212 - 213 - Note that you should add a newline at the end if setting a non-empty message 214 - since the knot won't do this for you.

-59

docs/migrations.md

··· 1 - # Migrations 2 - 3 - This document is laid out in reverse-chronological order. 4 - Newer migration guides are listed first, and older guides 5 - are further down the page. 6 - 7 - ## Upgrading from v1.8.x 8 - 9 - After v1.8.2, the HTTP API for knot and spindles have been 10 - deprecated and replaced with XRPC. Repositories on outdated 11 - knots will not be viewable from the appview. Upgrading is 12 - straightforward however. 13 - 14 - For knots: 15 - 16 - - Upgrade to latest tag (v1.9.0 or above) 17 - - Head to the [knot dashboard](https://tangled.org/settings/knots) and 18 - hit the "retry" button to verify your knot 19 - 20 - For spindles: 21 - 22 - - Upgrade to latest tag (v1.9.0 or above) 23 - - Head to the [spindle 24 - dashboard](https://tangled.org/settings/spindles) and hit the 25 - "retry" button to verify your spindle 26 - 27 - ## Upgrading from v1.7.x 28 - 29 - After v1.7.0, knot secrets have been deprecated. You no 30 - longer need a secret from the appview to run a knot. All 31 - authorized commands to knots are managed via [Inter-Service 32 - Authentication](https://atproto.com/specs/xrpc#inter-service-authentication-jwt). 33 - Knots will be read-only until upgraded. 34 - 35 - Upgrading is quite easy, in essence: 36 - 37 - - `KNOT_SERVER_SECRET` is no more, you can remove this 38 - environment variable entirely 39 - - `KNOT_SERVER_OWNER` is now required on boot, set this to 40 - your DID. You can find your DID in the 41 - [settings](https://tangled.org/settings) page. 42 - - Restart your knot once you have replaced the environment 43 - variable 44 - - Head to the [knot dashboard](https://tangled.org/settings/knots) and 45 - hit the "retry" button to verify your knot. This simply 46 - writes a `sh.tangled.knot` record to your PDS. 47 - 48 - If you use the nix module, simply bump the flake to the 49 - latest revision, and change your config block like so: 50 - 51 - ```diff 52 - services.tangled.knot = { 53 - enable = true; 54 - server = { 55 - - secretFile = /path/to/secret; 56 - + owner = "did:plc:foo"; 57 - }; 58 - }; 59 - ```

-25

docs/spindle/architecture.md

··· 1 - # spindle architecture 2 - 3 - Spindle is a small CI runner service. Here's a high level overview of how it operates: 4 - 5 - * listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and 6 - [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream. 7 - * when a new repo record comes through (typically when you add a spindle to a 8 - repo from the settings), spindle then resolves the underlying knot and 9 - subscribes to repo events (see: 10 - [`sh.tangled.pipeline`](/lexicons/pipeline.json)). 11 - * the spindle engine then handles execution of the pipeline, with results and 12 - logs beamed on the spindle event stream over wss 13 - 14 - ### the engine 15 - 16 - At present, the only supported backend is Docker (and Podman, if Docker 17 - compatibility is enabled, so that `/run/docker.sock` is created). Spindle 18 - executes each step in the pipeline in a fresh container, with state persisted 19 - across steps within the `/tangled/workspace` directory. 20 - 21 - The base image for the container is constructed on the fly using 22 - [Nixery](https://nixery.dev), which is handy for caching layers for frequently 23 - used packages. 24 - 25 - The pipeline manifest is [specified here](/docs/spindle/pipeline.md).

-52

docs/spindle/hosting.md

··· 1 - # spindle self-hosting guide 2 - 3 - ## prerequisites 4 - 5 - * Go 6 - * Docker (the only supported backend currently) 7 - 8 - ## configuration 9 - 10 - Spindle is configured using environment variables. The following environment variables are available: 11 - 12 - * `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`). 13 - * `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`). 14 - * `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required). 15 - * `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`). 16 - * `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`). 17 - * `SPINDLE_SERVER_OWNER`: The DID of the owner (required). 18 - * `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`). 19 - * `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`). 20 - * `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`). 21 - 22 - ## running spindle 23 - 24 - 1. **Set the environment variables.** For example: 25 - 26 - ```shell 27 - export SPINDLE_SERVER_HOSTNAME="your-hostname" 28 - export SPINDLE_SERVER_OWNER="your-did" 29 - ``` 30 - 31 - 2. **Build the Spindle binary.** 32 - 33 - ```shell 34 - cd core 35 - go mod download 36 - go build -o cmd/spindle/spindle cmd/spindle/main.go 37 - ``` 38 - 39 - 3. **Create the log directory.** 40 - 41 - ```shell 42 - sudo mkdir -p /var/log/spindle 43 - sudo chown $USER:$USER -R /var/log/spindle 44 - ``` 45 - 46 - 4. **Run the Spindle binary.** 47 - 48 - ```shell 49 - ./cmd/spindle/spindle 50 - ``` 51 - 52 - Spindle will now start, connect to the Jetstream server, and begin processing pipelines.

-285

docs/spindle/openbao.md

··· 1 - # spindle secrets with openbao 2 - 3 - This document covers setting up Spindle to use OpenBao for secrets 4 - management via OpenBao Proxy instead of the default SQLite backend. 5 - 6 - ## overview 7 - 8 - Spindle now uses OpenBao Proxy for secrets management. The proxy handles 9 - authentication automatically using AppRole credentials, while Spindle 10 - connects to the local proxy instead of directly to the OpenBao server. 11 - 12 - This approach provides better security, automatic token renewal, and 13 - simplified application code. 14 - 15 - ## installation 16 - 17 - Install OpenBao from nixpkgs: 18 - 19 - ```bash 20 - nix shell nixpkgs#openbao # for a local server 21 - ``` 22 - 23 - ## setup 24 - 25 - The setup process can is documented for both local development and production. 26 - 27 - ### local development 28 - 29 - Start OpenBao in dev mode: 30 - 31 - ```bash 32 - bao server -dev -dev-root-token-id="root" -dev-listen-address=127.0.0.1:8201 33 - ``` 34 - 35 - This starts OpenBao on `http://localhost:8201` with a root token. 36 - 37 - Set up environment for bao CLI: 38 - 39 - ```bash 40 - export BAO_ADDR=http://localhost:8200 41 - export BAO_TOKEN=root 42 - ``` 43 - 44 - ### production 45 - 46 - You would typically use a systemd service with a configuration file. Refer to 47 - [@tangled.org/infra](https://tangled.org/@tangled.org/infra) for how this can be 48 - achieved using Nix. 49 - 50 - Then, initialize the bao server: 51 - ```bash 52 - bao operator init -key-shares=1 -key-threshold=1 53 - ``` 54 - 55 - This will print out an unseal key and a root key. Save them somewhere (like a password manager). Then unseal the vault to begin setting it up: 56 - ```bash 57 - bao operator unseal <unseal_key> 58 - ``` 59 - 60 - All steps below remain the same across both dev and production setups. 61 - 62 - ### configure openbao server 63 - 64 - Create the spindle KV mount: 65 - 66 - ```bash 67 - bao secrets enable -path=spindle -version=2 kv 68 - ``` 69 - 70 - Set up AppRole authentication and policy: 71 - 72 - Create a policy file `spindle-policy.hcl`: 73 - 74 - ```hcl 75 - # Full access to spindle KV v2 data 76 - path "spindle/data/*" { 77 - capabilities = ["create", "read", "update", "delete"] 78 - } 79 - 80 - # Access to metadata for listing and management 81 - path "spindle/metadata/*" { 82 - capabilities = ["list", "read", "delete", "update"] 83 - } 84 - 85 - # Allow listing at root level 86 - path "spindle/" { 87 - capabilities = ["list"] 88 - } 89 - 90 - # Required for connection testing and health checks 91 - path "auth/token/lookup-self" { 92 - capabilities = ["read"] 93 - } 94 - ``` 95 - 96 - Apply the policy and create an AppRole: 97 - 98 - ```bash 99 - bao policy write spindle-policy spindle-policy.hcl 100 - bao auth enable approle 101 - bao write auth/approle/role/spindle \ 102 - token_policies="spindle-policy" \ 103 - token_ttl=1h \ 104 - token_max_ttl=4h \ 105 - bind_secret_id=true \ 106 - secret_id_ttl=0 \ 107 - secret_id_num_uses=0 108 - ``` 109 - 110 - Get the credentials: 111 - 112 - ```bash 113 - # Get role ID (static) 114 - ROLE_ID=$(bao read -field=role_id auth/approle/role/spindle/role-id) 115 - 116 - # Generate secret ID 117 - SECRET_ID=$(bao write -f -field=secret_id auth/approle/role/spindle/secret-id) 118 - 119 - echo "Role ID: $ROLE_ID" 120 - echo "Secret ID: $SECRET_ID" 121 - ``` 122 - 123 - ### create proxy configuration 124 - 125 - Create the credential files: 126 - 127 - ```bash 128 - # Create directory for OpenBao files 129 - mkdir -p /tmp/openbao 130 - 131 - # Save credentials 132 - echo "$ROLE_ID" > /tmp/openbao/role-id 133 - echo "$SECRET_ID" > /tmp/openbao/secret-id 134 - chmod 600 /tmp/openbao/role-id /tmp/openbao/secret-id 135 - ``` 136 - 137 - Create a proxy configuration file `/tmp/openbao/proxy.hcl`: 138 - 139 - ```hcl 140 - # OpenBao server connection 141 - vault { 142 - address = "http://localhost:8200" 143 - } 144 - 145 - # Auto-Auth using AppRole 146 - auto_auth { 147 - method "approle" { 148 - mount_path = "auth/approle" 149 - config = { 150 - role_id_file_path = "/tmp/openbao/role-id" 151 - secret_id_file_path = "/tmp/openbao/secret-id" 152 - } 153 - } 154 - 155 - # Optional: write token to file for debugging 156 - sink "file" { 157 - config = { 158 - path = "/tmp/openbao/token" 159 - mode = 0640 160 - } 161 - } 162 - } 163 - 164 - # Proxy listener for Spindle 165 - listener "tcp" { 166 - address = "127.0.0.1:8201" 167 - tls_disable = true 168 - } 169 - 170 - # Enable API proxy with auto-auth token 171 - api_proxy { 172 - use_auto_auth_token = true 173 - } 174 - 175 - # Enable response caching 176 - cache { 177 - use_auto_auth_token = true 178 - } 179 - 180 - # Logging 181 - log_level = "info" 182 - ``` 183 - 184 - ### start the proxy 185 - 186 - Start OpenBao Proxy: 187 - 188 - ```bash 189 - bao proxy -config=/tmp/openbao/proxy.hcl 190 - ``` 191 - 192 - The proxy will authenticate with OpenBao and start listening on 193 - `127.0.0.1:8201`. 194 - 195 - ### configure spindle 196 - 197 - Set these environment variables for Spindle: 198 - 199 - ```bash 200 - export SPINDLE_SERVER_SECRETS_PROVIDER=openbao 201 - export SPINDLE_SERVER_SECRETS_OPENBAO_PROXY_ADDR=http://127.0.0.1:8201 202 - export SPINDLE_SERVER_SECRETS_OPENBAO_MOUNT=spindle 203 - ``` 204 - 205 - Start Spindle: 206 - 207 - Spindle will now connect to the local proxy, which handles all 208 - authentication automatically. 209 - 210 - ## production setup for proxy 211 - 212 - For production, you'll want to run the proxy as a service: 213 - 214 - Place your production configuration in `/etc/openbao/proxy.hcl` with 215 - proper TLS settings for the vault connection. 216 - 217 - ## verifying setup 218 - 219 - Test the proxy directly: 220 - 221 - ```bash 222 - # Check proxy health 223 - curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/sys/health 224 - 225 - # Test token lookup through proxy 226 - curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/auth/token/lookup-self 227 - ``` 228 - 229 - Test OpenBao operations through the server: 230 - 231 - ```bash 232 - # List all secrets 233 - bao kv list spindle/ 234 - 235 - # Add a test secret via Spindle API, then check it exists 236 - bao kv list spindle/repos/ 237 - 238 - # Get a specific secret 239 - bao kv get spindle/repos/your_repo_path/SECRET_NAME 240 - ``` 241 - 242 - ## how it works 243 - 244 - - Spindle connects to OpenBao Proxy on localhost (typically port 8200 or 8201) 245 - - The proxy authenticates with OpenBao using AppRole credentials 246 - - All Spindle requests go through the proxy, which injects authentication tokens 247 - - Secrets are stored at `spindle/repos/{sanitized_repo_path}/{secret_key}` 248 - - Repository paths like `did:plc:alice/myrepo` become `did_plc_alice_myrepo` 249 - - The proxy handles all token renewal automatically 250 - - Spindle no longer manages tokens or authentication directly 251 - 252 - ## troubleshooting 253 - 254 - **Connection refused**: Check that the OpenBao Proxy is running and 255 - listening on the configured address. 256 - 257 - **403 errors**: Verify the AppRole credentials are correct and the policy 258 - has the necessary permissions. 259 - 260 - **404 route errors**: The spindle KV mount probably doesn't exist - run 261 - the mount creation step again. 262 - 263 - **Proxy authentication failures**: Check the proxy logs and verify the 264 - role-id and secret-id files are readable and contain valid credentials. 265 - 266 - **Secret not found after writing**: This can indicate policy permission 267 - issues. Verify the policy includes both `spindle/data/*` and 268 - `spindle/metadata/*` paths with appropriate capabilities. 269 - 270 - Check proxy logs: 271 - 272 - ```bash 273 - # If running as systemd service 274 - journalctl -u openbao-proxy -f 275 - 276 - # If running directly, check the console output 277 - ``` 278 - 279 - Test AppRole authentication manually: 280 - 281 - ```bash 282 - bao write auth/approle/login \ 283 - role_id="$(cat /tmp/openbao/role-id)" \ 284 - secret_id="$(cat /tmp/openbao/secret-id)" 285 - ```

-183

docs/spindle/pipeline.md

··· 1 - # spindle pipelines 2 - 3 - Spindle workflows allow you to write CI/CD pipelines in a simple format. They're located in the `.tangled/workflows` directory at the root of your repository, and are defined using YAML. 4 - 5 - The fields are: 6 - 7 - - [Trigger](#trigger): A **required** field that defines when a workflow should be triggered. 8 - - [Engine](#engine): A **required** field that defines which engine a workflow should run on. 9 - - [Clone options](#clone-options): An **optional** field that defines how the repository should be cloned. 10 - - [Dependencies](#dependencies): An **optional** field that allows you to list dependencies you may need. 11 - - [Environment](#environment): An **optional** field that allows you to define environment variables. 12 - - [Steps](#steps): An **optional** field that allows you to define what steps should run in the workflow. 13 - 14 - ## Trigger 15 - 16 - The first thing to add to a workflow is the trigger, which defines when a workflow runs. This is defined using a `when` field, which takes in a list of conditions. Each condition has the following fields: 17 - 18 - - `event`: This is a **required** field that defines when your workflow should run. It's a list that can take one or more of the following values: 19 - - `push`: The workflow should run every time a commit is pushed to the repository. 20 - - `pull_request`: The workflow should run every time a pull request is made or updated. 21 - - `manual`: The workflow can be triggered manually. 22 - - `branch`: Defines which branches the workflow should run for. If used with the `push` event, commits to the branch(es) listed here will trigger the workflow. If used with the `pull_request` event, updates to pull requests targeting the branch(es) listed here will trigger the workflow. This field has no effect with the `manual` event. Supports glob patterns using `*` and `**` (e.g., `main`, `develop`, `release-*`). Either `branch` or `tag` (or both) must be specified for `push` events. 23 - - `tag`: Defines which tags the workflow should run for. Only used with the `push` event - when tags matching the pattern(s) listed here are pushed, the workflow will trigger. This field has no effect with `pull_request` or `manual` events. Supports glob patterns using `*` and `**` (e.g., `v*`, `v1.*`, `release-**`). Either `branch` or `tag` (or both) must be specified for `push` events. 24 - 25 - For example, if you'd like to define a workflow that runs when commits are pushed to the `main` and `develop` branches, or when pull requests that target the `main` branch are updated, or manually, you can do so with: 26 - 27 - ```yaml 28 - when: 29 - - event: ["push", "manual"] 30 - branch: ["main", "develop"] 31 - - event: ["pull_request"] 32 - branch: ["main"] 33 - ``` 34 - 35 - You can also trigger workflows on tag pushes. For instance, to run a deployment workflow when tags matching `v*` are pushed: 36 - 37 - ```yaml 38 - when: 39 - - event: ["push"] 40 - tag: ["v*"] 41 - ``` 42 - 43 - You can even combine branch and tag patterns in a single constraint (the workflow triggers if either matches): 44 - 45 - ```yaml 46 - when: 47 - - event: ["push"] 48 - branch: ["main", "release-*"] 49 - tag: ["v*", "stable"] 50 - ``` 51 - 52 - ## Engine 53 - 54 - Next is the engine on which the workflow should run, defined using the **required** `engine` field. The currently supported engines are: 55 - 56 - - `nixery`: This uses an instance of [Nixery](https://nixery.dev) to run steps, which allows you to add [dependencies](#dependencies) from [Nixpkgs](https://github.com/NixOS/nixpkgs). You can search for packages on https://search.nixos.org, and there's a pretty good chance the package(s) you're looking for will be there. 57 - 58 - Example: 59 - 60 - ```yaml 61 - engine: "nixery" 62 - ``` 63 - 64 - ## Clone options 65 - 66 - When a workflow starts, the first step is to clone the repository. You can customize this behavior using the **optional** `clone` field. It has the following fields: 67 - 68 - - `skip`: Setting this to `true` will skip cloning the repository. This can be useful if your workflow is doing something that doesn't require anything from the repository itself. This is `false` by default. 69 - - `depth`: This sets the number of commits, or the "clone depth", to fetch from the repository. For example, if you set this to 2, the last 2 commits will be fetched. By default, the depth is set to 1, meaning only the most recent commit will be fetched, which is the commit that triggered the workflow. 70 - - `submodules`: If you use [git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) in your repository, setting this field to `true` will recursively fetch all submodules. This is `false` by default. 71 - 72 - The default settings are: 73 - 74 - ```yaml 75 - clone: 76 - skip: false 77 - depth: 1 78 - submodules: false 79 - ``` 80 - 81 - ## Dependencies 82 - 83 - Usually when you're running a workflow, you'll need additional dependencies. The `dependencies` field lets you define which dependencies to get, and from where. It's a key-value map, with the key being the registry to fetch dependencies from, and the value being the list of dependencies to fetch. 84 - 85 - Say you want to fetch Node.js and Go from `nixpkgs`, and a package called `my_pkg` you've made from your own registry at your repository at `https://tangled.sh/@example.com/my_pkg`. You can define those dependencies like so: 86 - 87 - ```yaml 88 - dependencies: 89 - # nixpkgs 90 - nixpkgs: 91 - - nodejs 92 - - go 93 - # custom registry 94 - git+https://tangled.org/@example.com/my_pkg: 95 - - my_pkg 96 - ``` 97 - 98 - Now these dependencies are available to use in your workflow! 99 - 100 - ## Environment 101 - 102 - The `environment` field allows you define environment variables that will be available throughout the entire workflow. **Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.** 103 - 104 - Example: 105 - 106 - ```yaml 107 - environment: 108 - GOOS: "linux" 109 - GOARCH: "arm64" 110 - NODE_ENV: "production" 111 - MY_ENV_VAR: "MY_ENV_VALUE" 112 - ``` 113 - 114 - ## Steps 115 - 116 - The `steps` field allows you to define what steps should run in the workflow. It's a list of step objects, each with the following fields: 117 - 118 - - `name`: This field allows you to give your step a name. This name is visible in your workflow runs, and is used to describe what the step is doing. 119 - - `command`: This field allows you to define a command to run in that step. The step is run in a Bash shell, and the logs from the command will be visible in the pipelines page on the Tangled website. The [dependencies](#dependencies) you added will be available to use here. 120 - - `environment`: Similar to the global [environment](#environment) config, this **optional** field is a key-value map that allows you to set environment variables for the step. **Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.** 121 - 122 - Example: 123 - 124 - ```yaml 125 - steps: 126 - - name: "Build backend" 127 - command: "go build" 128 - environment: 129 - GOOS: "darwin" 130 - GOARCH: "arm64" 131 - - name: "Build frontend" 132 - command: "npm run build" 133 - environment: 134 - NODE_ENV: "production" 135 - ``` 136 - 137 - ## Complete workflow 138 - 139 - ```yaml 140 - # .tangled/workflows/build.yml 141 - 142 - when: 143 - - event: ["push", "manual"] 144 - branch: ["main", "develop"] 145 - - event: ["pull_request"] 146 - branch: ["main"] 147 - 148 - engine: "nixery" 149 - 150 - # using the default values 151 - clone: 152 - skip: false 153 - depth: 1 154 - submodules: false 155 - 156 - dependencies: 157 - # nixpkgs 158 - nixpkgs: 159 - - nodejs 160 - - go 161 - # custom registry 162 - git+https://tangled.org/@example.com/my_pkg: 163 - - my_pkg 164 - 165 - environment: 166 - GOOS: "linux" 167 - GOARCH: "arm64" 168 - NODE_ENV: "production" 169 - MY_ENV_VAR: "MY_ENV_VALUE" 170 - 171 - steps: 172 - - name: "Build backend" 173 - command: "go build" 174 - environment: 175 - GOOS: "darwin" 176 - GOARCH: "arm64" 177 - - name: "Build frontend" 178 - command: "npm run build" 179 - environment: 180 - NODE_ENV: "production" 181 - ``` 182 - 183 - If you want another example of a workflow, you can look at the one [Tangled uses to build the project](https://tangled.sh/@tangled.sh/core/blob/master/.tangled/workflows/build.yml).