docs: document webhooks #1074

+272 -111

1 changed file

Diff round #4

expand all

unified split

docs

DOCS.md

+272 -111

docs/DOCS.md

··· 3 3 author: The Tangled Contributors 4 4 date: 21 Sun, Dec 2025 5 5 abstract: | 6 - Tangled is a decentralized code hosting and collaboration 7 - platform. Every component of Tangled is open-source and 8 - self-hostable. [tangled.org](https://tangled.org) also 9 - provides hosting and CI services that are free to use. 10 - 11 - There are several models for decentralized code 12 - collaboration platforms, ranging from ActivityPub’s 13 - (Forgejo) federated model, to Radicle’s entirely P2P model. 14 - Our approach attempts to be the best of both worlds by 15 - adopting the AT Protocol—a protocol for building decentralized 16 - social applications with a central identity 17 - 18 - Our approach to this is the idea of “knots”. Knots are 19 - lightweight, headless servers that enable users to host Git 20 - repositories with ease. Knots are designed for either single 21 - or multi-tenant use which is perfect for self-hosting on a 22 - Raspberry Pi at home, or larger “community” servers. By 23 - default, Tangled provides managed knots where you can host 24 - your repositories for free. 25 - 26 - The appview at tangled.org acts as a consolidated "view" 27 - into the whole network, allowing users to access, clone and 28 - contribute to repositories hosted across different knots 29 - seamlessly. 6 + Tangled is a decentralized code hosting and collaboration 7 + platform. Every component of Tangled is open-source and 8 + self-hostable. [tangled.org](https://tangled.org) also 9 + provides hosting and CI services that are free to use. 10 + 11 + There are several models for decentralized code 12 + collaboration platforms, ranging from ActivityPub’s 13 + (Forgejo) federated model, to Radicle’s entirely P2P model. 14 + Our approach attempts to be the best of both worlds by 15 + adopting the AT Protocol—a protocol for building decentralized 16 + social applications with a central identity 17 + 18 + Our approach to this is the idea of “knots”. Knots are 19 + lightweight, headless servers that enable users to host Git 20 + repositories with ease. Knots are designed for either single 21 + or multi-tenant use which is perfect for self-hosting on a 22 + Raspberry Pi at home, or larger “community” servers. By 23 + default, Tangled provides managed knots where you can host 24 + your repositories for free. 25 + 26 + The appview at tangled.org acts as a consolidated "view" 27 + into the whole network, allowing users to access, clone and 28 + contribute to repositories hosted across different knots 29 + seamlessly. 30 30 --- 31 31 32 32 # Quick start guide ··· 131 131 cd my-project 132 132 133 133 git init 134 - echo "# My Project" > README.md 134 + echo "# My Project" > README.md 135 135 ``` 136 136 137 137 Add some content and push! ··· 313 313 and operation tool. For the purpose of this guide, we're 314 314 only concerned with these subcommands: 315 315 316 - * `knot server`: the main knot server process, typically 317 - run as a supervised service 318 - * `knot guard`: handles role-based access control for git 319 - over SSH (you'll never have to run this yourself) 320 - * `knot keys`: fetches SSH keys associated with your knot; 321 - we'll use this to generate the SSH 322 - `AuthorizedKeysCommand` 316 + - `knot server`: the main knot server process, typically 317 + run as a supervised service 318 + - `knot guard`: handles role-based access control for git 319 + over SSH (you'll never have to run this yourself) 320 + - `knot keys`: fetches SSH keys associated with your knot; 321 + we'll use this to generate the SSH 322 + `AuthorizedKeysCommand` 323 323 324 324 ``` 325 325 cd core ··· 432 432 can move these paths if you'd like to store them in another folder. Be careful 433 433 when adjusting these paths: 434 434 435 - * Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent 436 - any possible side effects. Remember to restart it once you're done. 437 - * Make backups before moving in case something goes wrong. 438 - * Make sure the `git` user can read and write from the new paths. 435 + - Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent 436 + any possible side effects. Remember to restart it once you're done. 437 + - Make backups before moving in case something goes wrong. 438 + - Make sure the `git` user can read and write from the new paths. 439 439 440 440 #### Database 441 441 ··· 519 519 2. Check to see that your knot has synced the key by running 520 520 `knot keys` 521 521 3. Check to see if git is supplying the correct private key 522 - when pushing: `GIT_SSH_COMMAND="ssh -v" git push ...` 522 + when pushing: `GIT_SSH_COMMAND="ssh -v" git push ...` 523 523 4. Check to see if `sshd` on the knot is rejecting the push 524 524 for some reason: `journalctl -xeu ssh` (or `sshd`, 525 525 depending on your machine). These logs are unavailable if ··· 527 527 5. Check to see if the knot itself is rejecting the push, 528 528 depending on your setup, the logs might be in one of the 529 529 following paths: 530 - * `/tmp/knotguard.log` 531 - * `/home/git/log` 532 - * `/home/git/guard.log` 530 + - `/tmp/knotguard.log` 531 + - `/home/git/log` 532 + - `/home/git/guard.log` 533 533 534 534 # Spindles 535 535 ··· 847 847 848 848 ### Prerequisites 849 849 850 - * Go 851 - * Docker (the only supported backend currently) 850 + - Go 851 + - Docker (the only supported backend currently) 852 852 853 853 ### Configuration 854 854 855 855 Spindle is configured using environment variables. The following environment variables are available: 856 856 857 - * `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`). 858 - * `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`). 859 - * `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required). 860 - * `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`). 861 - * `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`). 862 - * `SPINDLE_SERVER_OWNER`: The DID of the owner (required). 863 - * `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`). 864 - * `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`). 865 - * `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`). 857 + - `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`). 858 + - `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`). 859 + - `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required). 860 + - `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`). 861 + - `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`). 862 + - `SPINDLE_SERVER_OWNER`: The DID of the owner (required). 863 + - `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`). 864 + - `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`). 865 + - `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`). 866 866 867 867 ### Running spindle 868 868 869 - 1. **Set the environment variables.** For example: 869 + 1. **Set the environment variables.** For example: 870 870 871 871 ```shell 872 872 export SPINDLE_SERVER_HOSTNAME="your-hostname" ··· 900 900 901 901 Spindle is a small CI runner service. Here's a high-level overview of how it operates: 902 902 903 - * Listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and 904 - [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream. 905 - * When a new repo record comes through (typically when you add a spindle to a 906 - repo from the settings), spindle then resolves the underlying knot and 907 - subscribes to repo events (see: 908 - [`sh.tangled.pipeline`](/lexicons/pipeline.json)). 909 - * The spindle engine then handles execution of the pipeline, with results and 910 - logs beamed on the spindle event stream over WebSocket 903 + - Listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and 904 + [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream. 905 + - When a new repo record comes through (typically when you add a spindle to a 906 + repo from the settings), spindle then resolves the underlying knot and 907 + subscribes to repo events (see: 908 + [`sh.tangled.pipeline`](/lexicons/pipeline.json)). 909 + - The spindle engine then handles execution of the pipeline, with results and 910 + logs beamed on the spindle event stream over WebSocket 911 911 912 912 ### The engine 913 913 ··· 1221 1221 secret_id="$(cat /tmp/openbao/secret-id)" 1222 1222 ``` 1223 1223 1224 + # Webhooks 1225 + 1226 + Webhooks allow you to receive HTTP POST notifications when events occur in your repositories. This enables you to integrate Tangled with external services, trigger CI/CD pipelines, send notifications, or automate workflows. 1227 + 1228 + ## Overview 1229 + 1230 + Webhooks send HTTP POST requests to URLs you configure whenever specific events happen. Currently, Tangled supports push events, with more event types coming soon. 1231 + 1232 + ## Configuring webhooks 1233 + 1234 + To set up a webhook for your repository: 1235 + 1236 + 1. Navigate to your repository settings 1237 + 2. Click the "hooks" tab 1238 + 3. Click "add webhook" 1239 + 4. Configure your webhook: 1240 + - **Payload URL**: The endpoint that will receive the webhook POST requests 1241 + - **Secret**: An optional secret key for verifying webhook authenticity (auto-generated if left blank) 1242 + - **Events**: Select which events trigger the webhook (currently only push events) 1243 + - **Active**: Toggle whether the webhook is enabled 1244 + 1245 + ## Webhook payload 1246 + 1247 + ### Push 1248 + 1249 + When a push event occurs, Tangled sends a POST request with a JSON payload of the format: 1250 + 1251 + ```json 1252 + { 1253 + "after": "7b320e5cbee2734071e4310c1d9ae401d8f6cab5", 1254 + "before": "c04ddf64eddc90e4e2a9846ba3b43e67a0e2865e", 1255 + "pusher": { 1256 + "did": "did:plc:hwevmowznbiukdf6uk5dwrrq" 1257 + }, 1258 + "ref": "refs/heads/main", 1259 + "repository": { 1260 + "clone_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1261 + "created_at": "2025-09-15T08:57:23Z", 1262 + "description": "an example repository", 1263 + "fork": false, 1264 + "full_name": "did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1265 + "html_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1266 + "name": "some-repo", 1267 + "open_issues_count": 5, 1268 + "owner": { 1269 + "did": "did:plc:hwevmowznbiukdf6uk5dwrrq" 1270 + }, 1271 + "ssh_url": "ssh://git@tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo", 1272 + "stars_count": 1, 1273 + "updated_at": "2025-09-15T08:57:23Z" 1274 + } 1275 + } 1276 + ``` 1277 + 1278 + ## HTTP headers 1279 + 1280 + Each webhook request includes the following headers: 1281 + 1282 + - `Content-Type: application/json` 1283 + - `User-Agent: Tangled-Hook/<short-sha>` — User agent with short SHA of the commit 1284 + - `X-Tangled-Event: push` — The event type 1285 + - `X-Tangled-Hook-ID: <webhook-id>` — The webhook ID 1286 + - `X-Tangled-Delivery: <uuid>` — Unique delivery ID 1287 + - `X-Tangled-Signature-256: sha256=<hmac>` — HMAC-SHA256 signature (if secret configured) 1288 + 1289 + ## Verifying webhook signatures 1290 + 1291 + If you configured a secret, you should verify the webhook signature to ensure requests are authentic. For example, in Go: 1292 + 1293 + ```go 1294 + package main 1295 + 1296 + import ( 1297 + "crypto/hmac" 1298 + "crypto/sha256" 1299 + "encoding/hex" 1300 + "io" 1301 + "net/http" 1302 + "strings" 1303 + ) 1304 + 1305 + func verifySignature(payload []byte, signatureHeader, secret string) bool { 1306 + // Remove 'sha256=' prefix from signature header 1307 + signature := strings.TrimPrefix(signatureHeader, "sha256=") 1308 + 1309 + // Compute expected signature 1310 + mac := hmac.New(sha256.New, []byte(secret)) 1311 + mac.Write(payload) 1312 + expected := hex.EncodeToString(mac.Sum(nil)) 1313 + 1314 + // Use constant-time comparison to prevent timing attacks 1315 + return hmac.Equal([]byte(signature), []byte(expected)) 1316 + } 1317 + 1318 + func webhookHandler(w http.ResponseWriter, r *http.Request) { 1319 + // Read the request body 1320 + payload, err := io.ReadAll(r.Body) 1321 + if err != nil { 1322 + http.Error(w, "Bad request", http.StatusBadRequest) 1323 + return 1324 + } 1325 + 1326 + // Get signature from header 1327 + signatureHeader := r.Header.Get("X-Tangled-Signature-256") 1328 + 1329 + // Verify signature 1330 + if signatureHeader != "" && verifySignature(payload, signatureHeader, yourSecret) { 1331 + // Webhook is authentic, process it 1332 + processWebhook(payload) 1333 + w.WriteHeader(http.StatusOK) 1334 + } else { 1335 + http.Error(w, "Invalid signature", http.StatusUnauthorized) 1336 + } 1337 + } 1338 + ``` 1339 + 1340 + ## Delivery retries 1341 + 1342 + Webhooks are automatically retried on failure: 1343 + 1344 + - **3 total attempts** (1 initial + 2 retries) 1345 + - **Exponential backoff** starting at 1 second, max 10 seconds 1346 + - **Retried on**: 1347 + - Network errors 1348 + - HTTP 5xx server errors 1349 + - **Not retried on**: 1350 + - HTTP 4xx client errors (bad request, unauthorized, etc.) 1351 + 1352 + ### Timeouts 1353 + 1354 + Webhook requests timeout after 30 seconds. If your endpoint needs more time: 1355 + 1356 + 1. Respond with 200 OK immediately 1357 + 2. Process the webhook asynchronously in the background 1358 + 1359 + ## Example integrations 1360 + 1361 + ### Discord notifications 1362 + 1363 + ```javascript 1364 + app.post("/webhook", (req, res) => { 1365 + const payload = req.body; 1366 + 1367 + fetch("https://discord.com/api/webhooks/...", { 1368 + method: "POST", 1369 + headers: { "Content-Type": "application/json" }, 1370 + body: JSON.stringify({ 1371 + content: `New push to ${payload.repository.full_name}`, 1372 + embeds: [ 1373 + { 1374 + title: `${payload.pusher.did} pushed to ${payload.ref}`, 1375 + url: payload.repository.html_url, 1376 + color: 0x00ff00, 1377 + }, 1378 + ], 1379 + }), 1380 + }); 1381 + 1382 + res.status(200).send("OK"); 1383 + }); 1384 + ``` 1385 + 1224 1386 # Migrating knots and spindles 1225 1387 1226 1388 Sometimes, non-backwards compatible changes are made to the ··· 1356 1518 <details> 1357 1519 <summary><strong>macOS users will have to set up a Nix Builder first</strong></summary> 1358 1520 1359 - In order to build Tangled's dev VM on macOS, you will 1360 - first need to set up a Linux Nix builder. The recommended 1361 - way to do so is to run a [`darwin.linux-builder` 1362 - VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder) 1363 - and to register it in `nix.conf` as a builder for Linux 1364 - with the same architecture as your Mac (`linux-aarch64` if 1365 - you are using Apple Silicon). 1366 - 1367 - > IMPORTANT: You must build `darwin.linux-builder` somewhere other than inside 1368 - > the Tangled repo so that it doesn't conflict with the other VM. For example, 1369 - > you can do 1370 - > 1371 - > ```shell 1372 - > cd $(mktemp -d buildervm.XXXXX) && nix run nixpkgs#darwin.linux-builder 1373 - > ``` 1374 - > 1375 - > to store the builder VM in a temporary dir. 1376 - > 1377 - > You should read and follow [all the other intructions][darwin builder vm] to 1378 - > avoid subtle problems. 1379 - 1380 - Alternatively, you can use any other method to set up a 1381 - Linux machine with Nix installed that you can `sudo ssh` 1382 - into (in other words, root user on your Mac has to be able 1383 - to ssh into the Linux machine without entering a password) 1384 - and that has the same architecture as your Mac. See 1385 - [remote builder 1386 - instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements) 1387 - for how to register such a builder in `nix.conf`. 1388 - 1389 - > WARNING: If you'd like to use 1390 - > [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or 1391 - > [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo 1392 - > ssh` works can be tricky. It seems to be [possible with 1393 - > Orbstack](https://github.com/orgs/orbstack/discussions/1669). 1521 + In order to build Tangled's dev VM on macOS, you will 1522 + first need to set up a Linux Nix builder. The recommended 1523 + way to do so is to run a [`darwin.linux-builder` 1524 + VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder) 1525 + and to register it in `nix.conf` as a builder for Linux 1526 + with the same architecture as your Mac (`linux-aarch64` if 1527 + you are using Apple Silicon). 1528 + 1529 + > IMPORTANT: You must build `darwin.linux-builder` somewhere other than inside 1530 + > the Tangled repo so that it doesn't conflict with the other VM. For example, 1531 + > you can do 1532 + > 1533 + > ```shell 1534 + > cd $(mktemp -d buildervm.XXXXX) && nix run nixpkgs#darwin.linux-builder 1535 + > ``` 1536 + > 1537 + > to store the builder VM in a temporary dir. 1538 + > 1539 + > You should read and follow [all the other intructions][darwin builder vm] to 1540 + > avoid subtle problems. 1541 + 1542 + Alternatively, you can use any other method to set up a 1543 + Linux machine with Nix installed that you can `sudo ssh` 1544 + into (in other words, root user on your Mac has to be able 1545 + to ssh into the Linux machine without entering a password) 1546 + and that has the same architecture as your Mac. See 1547 + [remote builder 1548 + instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements) 1549 + for how to register such a builder in `nix.conf`. 1550 + 1551 + > WARNING: If you'd like to use 1552 + > [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or 1553 + > [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo 1554 + ssh` works can be tricky. It seems to be [possible with 1555 + > Orbstack](https://github.com/orgs/orbstack/discussions/1669). 1394 1556 1395 1557 </details> 1396 1558 ··· 1463 1625 1464 1626 We follow a commit style similar to the Go project. Please keep commits: 1465 1627 1466 - * **atomic**: each commit should represent one logical change 1467 - * **descriptive**: the commit message should clearly describe what the 1468 - change does and why it's needed 1628 + - **atomic**: each commit should represent one logical change 1629 + - **descriptive**: the commit message should clearly describe what the 1630 + change does and why it's needed 1469 1631 1470 1632 ### Message format 1471 1633 ··· 1491 1653 knotserver/git/service: improve error checking in upload-pack 1492 1654 ``` 1493 1655 1494 - 1495 1656 ### General notes 1496 1657 1497 1658 - PRs get merged "as-is" (fast-forward)—like applying a patch-series 1498 - using `git am`. At present, there is no squashing—so please author 1499 - your commits as they would appear on `master`, following the above 1500 - guidelines. 1659 + using `git am`. At present, there is no squashing—so please author 1660 + your commits as they would appear on `master`, following the above 1661 + guidelines. 1501 1662 - If there is a lot of nesting, for example "appview: 1502 - pages/templates/repo/fragments: ...", these can be truncated down to 1503 - just "appview: repo/fragments: ...". If the change affects a lot of 1504 - subdirectories, you may abbreviate to just the top-level names, e.g. 1505 - "appview: ..." or "knotserver: ...". 1663 + pages/templates/repo/fragments: ...", these can be truncated down to 1664 + just "appview: repo/fragments: ...". If the change affects a lot of 1665 + subdirectories, you may abbreviate to just the top-level names, e.g. 1666 + "appview: ..." or "knotserver: ...". 1506 1667 - Keep commits lowercased with no trailing period. 1507 1668 - Use the imperative mood in the summary line (e.g., "fix bug" not 1508 - "fixed bug" or "fixes bug"). 1669 + "fixed bug" or "fixes bug"). 1509 1670 - Try to keep the summary line under 72 characters, but we aren't too 1510 - fussed about this. 1671 + fussed about this. 1511 1672 - Follow the same formatting for PR titles if filled manually. 1512 1673 - Don't include unrelated changes in the same commit. 1513 1674 - Avoid noisy commit messages like "wip" or "final fix"—rewrite history 1514 - before submitting if necessary. 1675 + before submitting if necessary. 1515 1676 1516 1677 ## Code formatting 1517 1678 ··· 1601 1762 1602 1763 - You may need to ensure that your PDS is timesynced using 1603 1764 NTP: 1604 - * Enable the `ntpd` service 1605 - * Run `ntpd -qg` to synchronize your clock 1765 + - Enable the `ntpd` service 1766 + - Run `ntpd -qg` to synchronize your clock 1606 1767 - You may need to increase the default request timeout: 1607 1768 `NODE_OPTIONS="--network-family-autoselection-attempt-timeout=500"` 1608 1769