# treadle workflows A workflow is a YAML file in `.tangled/workflows/` of a repo. treadle reads every regular file in that directory whose name ends in `.yml` or `.yaml`. The file name is the name of the workflow. treadle skips a workflow whose `engine` it does not know. Each event that matches one or more workflows makes one pipeline. The pipeline has one run for each matching workflow. The runs of a pipeline run in parallel, unless `treadle.after` orders them. A file that treadle cannot parse makes a failed run, with the parse error as its error. treadle matches the run to events by the `when` of the file. When treadle cannot read the YAML or the `when` of the file, the run matches every event. A failed run of a pull request does not wait for Approve. treadle reads at most 50 workflow files, each of at most 256 KiB. When a commit has more files or a larger file, a push or a manual run makes one failed run named `.tangled/workflows`, and a pull request gets no pipeline. ## Engines The `engine` key selects where the workflow runs: - `engine: buildkite` starts a build of a Buildkite pipeline. Buildkite runs the steps of that pipeline. treadle ignores `steps`, `environment`, `dependencies`, `secrets`, `cache` and `runs_on` in the workflow file. - `engine: cloudflare` runs the steps of the workflow file in a Cloudflare container. treadle starts one container for each run. treadle skips a workflow with any other engine, or with no engine. ## Triggers `when` is a list of conditions. A workflow matches an event when one condition matches. Each field takes a string or a list of strings: - `event`: `push`, `pull_request` or `manual`. - `branch`: patterns for the branch of a push, or for the target branch of a pull request. - `tag`: patterns for the tag of a push. - `types`: the actions of a pull request that start a run. The default is `opened`, `reopened` and `synchronize`. treadle sends `opened` for the first patch of a pull request and `synchronize` for each later patch. A workflow without `when` matches every push and every pull request. ```yaml when: - event: push branch: [main, "release/*"] - event: push tag: "v*" - event: pull_request branch: main - event: manual ``` Rules for each event: - A push to a branch matches a condition with `event: push` and a matching `branch` pattern. A push of a tag needs a matching `tag` pattern. A push of any other ref, or a push that deletes a ref, starts no run. - A push with the push option `skip-ci` or `ci-skip` starts no run. - A pull request matches a condition with `event: pull_request`, a `branch` pattern that matches the target branch, and a matching action. - A manual run ignores `when`. It runs every workflow of the commit, or only the workflows that it names and the workflows that they wait for with `treadle.after`. ### Patterns - `*` matches any characters except `/`. - `**` matches across `/` when it is a whole path segment, for example `**` or `release/**`. - `?` matches one character except `/`. - `[abc]` matches one of the characters. `[!abc]` matches any other character. ### Manual inputs A manual run can carry inputs, each with a name and a value. An input name has only letters, digits and underscores, and does not start with a digit. For `engine: buildkite`, the build gets each input as the variable `TANGLED_INPUT_`, with the name in upper case. `engine: cloudflare` runs do not get the inputs. ## Steps Only `engine: cloudflare` reads `steps`. It is a list that must not be empty. Each step needs a `name` and a `command`. ```yaml engine: cloudflare steps: - name: Install command: npm ci - name: Test command: | npm test npm run lint ``` The steps run in order in `/workspace`, which holds the files of the commit without the Git history. Each command runs with `bash -eo pipefail -c`. The first step that fails stops the run, and the run fails. The container uses the `cloudflare/debian-trixie` image and has internet access. ## Environment Only `engine: cloudflare` reads `environment`. Set it for the whole workflow, for one step, or both. A step variable replaces a workflow variable with the same name. ```yaml environment: NODE_ENV: test PYTHON_VERSION: 3.10 steps: - name: Test command: npm test environment: DEBUG: "1" ``` Rules: - `environment` is a map. A name has letters, digits and underscores, and does not start with a digit. - A value is a string, a number or a boolean. treadle keeps a number as written, so `3.10` stays `3.10`. - The map must not come from a YAML alias. Each step also gets these variables: - `CI=true`, `HOME=/root`, and a standard `PATH`. - `TREADLE_REPO`, `TREADLE_COMMIT`, `TREADLE_PIPELINE` and `TREADLE_WORKFLOW`. - The secrets of the workflow, when the run is trusted. An injected secret holds a placeholder, and `NODE_EXTRA_CA_CERTS` is set. An `engine: buildkite` build gets `TREADLE_REPO`, `TREADLE_PIPELINE` and `TREADLE_WORKFLOW` as build variables, and the same names as build meta-data with the prefix `treadle:`. ## Dependencies Only `engine: cloudflare` reads `dependencies`. It is a flat list of Debian package names. A name can include a version, as in `curl=8.14.1-2`. ```yaml dependencies: [ca-certificates, git, python3] ``` treadle installs the packages with `apt-get` in a first step called `Install dependencies`. This step shows in the log, and it fails the run as other steps do. ## Secrets Only `engine: cloudflare` reads `secrets`. It lists the repo secrets that the steps get as variables. An item is a secret name, or a map that injects the secret into HTTPS requests. ```yaml secrets: - NPM_TOKEN - name: CLOUDFLARE_API_TOKEN hosts: [api.cloudflare.com] - name: OTHER_KEY hosts: ["*.example.com"] headers: [X-Auth-Key] ``` Rules: - Only a trusted run gets secrets. An untrusted run gets none, and the variables are not set. - A trusted run fails when a secret that it names is not set. - treadle replaces each secret value in the log with `***`. A log line with a secret loses its colors. - A name appears only once in the list. - The Tangled web app sets secrets with the `org.tangled.secret.*` methods. A secret name is a variable name of at most 50 characters. A value has 1 to 4096 bytes and no NUL character. `engine: buildkite` builds get no secrets from treadle. Keep the secrets of a Buildkite build in Buildkite. ### Injected secrets The value of an injected secret is not in the container. The variable holds a random placeholder, which is new for each run. treadle intercepts the HTTPS connections to the `hosts` of the secret. In a request to one of these hosts, treadle replaces the placeholder with the value in the `headers` of the secret. The URL and the body do not change. - `hosts` lists at most 64 lowercase hostnames on port 443, not IP addresses. `*.example.com` matches each subdomain of `example.com`, but not `example.com`. - `headers` lists the header names. The default is `[Authorization]`. In a `Basic` value, treadle also replaces the placeholder in the decoded `user:password`. - treadle replaces the placeholder only when the URL of the request has one of the hosts and port 443. A request to another host keeps the placeholder. - treadle does not follow redirects. The client in the container gets the redirect. - Node.js trusts the intercept CA through `NODE_EXTRA_CA_CERTS`. treadle also adds the CA to the system store when `update-ca-certificates` is installed, for example with `dependencies: [ca-certificates]`. Other clients may need their own setting, for example `REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt` for Python `requests`. - A secret that is not an HTTP header, such as an SSH key or a signing key, must be a plain name. Injection stops only the theft of the stored value. For example, a dependency that sends the environment to another host gets only the placeholder, and the files of the container never hold the value. Injection does not stop the use of the secret: during a trusted run, each step can do on its hosts all that the secret allows, for example create a new token. treadle does not filter or mask the responses, so a run with injected secrets saves the trusted snapshot. Rules for `hosts`: - Never list a host that returns request headers in its response, because a step can then read the value. - Never list a host or glob where other people can create names, such as `*.workers.dev` or `*.github.io`. ## Trust A trusted run of an `engine: cloudflare` workflow gets its secrets and can use the trusted cache. treadle decides trust as follows: - A push to a trusted branch is trusted. By default, only the default branch is trusted. - A manual run on a trusted branch is trusted only when its commit is the head of that branch. - A push of a tag is not trusted. - A pull request run is never trusted, whatever its branch. - A retry keeps the trust of the run that it repeats. `trusted.branch` lists the trusted branches as patterns. treadle reads it only from the workflow file on the default branch, so a branch or a pull request cannot make itself trusted. ```yaml trusted: branch: [main, "release/*"] ``` ### Pull requests and Approve treadle finds new patches of pull requests on the repos that it serves. The runs of a pull request start at once when the repo owner opened the pull request and sent the patch. The runs of any other pull request wait for Approve in the Tangled web app. A retry of a pull request that did not get Approve also waits for Approve. For each repo, treadle keeps at most 10 pull request pipelines from one sender that wait for Approve. A new patch from that sender deletes the oldest of them. treadle deletes a pipeline that waits for Approve for 14 days. ## Cache and snapshots `engine: cloudflare` runs can start from a snapshot of an earlier container. Set `cache: true` to turn this on. It is off by default. ```yaml cache: true ``` After a successful trusted run at the head of the default branch, treadle removes `/workspace`, its own files and `/tmp`, and saves a snapshot of the container. The snapshot keeps the whole root file system, so it also keeps every file that a step or a dependency changed outside those directories. This shows as a last step called `Save cache`. A failed snapshot does not fail the run. treadle keeps up to two snapshots for each workflow: - The trusted snapshot comes from a run that had secrets, or that started from the trusted snapshot. Only trusted runs start from it. - The public snapshot comes from any other trusted run. Every run can start from it. A newer public snapshot replaces an older trusted snapshot, so trusted runs then start from the public snapshot. When a snapshot does not start, the run starts again from the image. A run starts from a snapshot only when its workflow file is the same as the file of the run that saved the snapshot. After a change to the workflow file, runs start from the image again. After a successful trusted run with `cache: false` at the head of the default branch, treadle discards the saved snapshots of the workflow. ## Instance types and timeouts `runs_on` sets the container size. The default is `standard-1`. Give the name of an [instance type](https://developers.cloudflare.com/containers/platform/limits/#instance-types), such as `lite`, `basic` or `standard-2`, or a [custom size](https://developers.cloudflare.com/containers/platform/limits/#custom-instance-types) with `vcpu`, `memory_mib` and `disk_mb`. Cloudflare Containers checks the values when the container starts, and the run fails if they are not valid. `treadle.cloudflare.timeout` sets the time limit of a run. The default is `1h`, and the maximum is `6h`. Write it with hours, minutes and seconds, as in `1h30m`, `45m` or `90s`. A run that is still running at the limit ends with the status `timeout`. `timeout` is the only key of `treadle.cloudflare`. treadle rejects a workflow with any other key there. ```yaml runs_on: standard-2 treadle: cloudflare: timeout: 20m ``` ```yaml runs_on: vcpu: 2 memory_mib: 8192 disk_mb: 16000 ``` ## Buildkite settings `treadle.buildkite` sets the Buildkite pipeline of an `engine: buildkite` workflow: - `pipeline`: the pipeline slug. The default is the repo name in lower case, with each run of other characters replaced by `-`, and with no `-` at the start or end. treadle skips the workflow when this default is empty. - `org`: the Buildkite organization. The default is the organization of the treadle deploy. - `clean_checkout`: `true` or `false`, sent to Buildkite as the clean checkout setting of the build. A pull request run takes `pipeline` and `org` from the file of the same name on the default branch, so a pull request cannot choose the pipeline that builds it. When the default branch has no such file, the run uses the defaults. ```yaml engine: buildkite when: - event: push branch: main treadle: buildkite: pipeline: my-app clean_checkout: true ``` The build gets the commit, the branch, the commit message and author, and for a pull request the target branch. Buildkite branch filters of the pipeline do not apply. ## Order `treadle.after` lists the workflows of the same pipeline that must succeed before the run starts. A name can leave out `.yml` or `.yaml`, so `check` names `check.yml`. It works for both engines. A waiting run has the status `pending`. It has no container and no Buildkite build, and it does not count toward the limit of 10 runs. ```yaml engine: cloudflare secrets: [CLOUDFLARE_API_TOKEN] treadle: after: [check.yml] ``` Rules: - The run starts when every workflow that it names has the status `success`. - When one of them fails, times out or is cancelled, or its file is not valid, the run is cancelled with an error that names it. - A name that is not a workflow of the pipeline, or a list that makes a cycle, fails the run at once. A workflow that does not match the event is not in the pipeline, so a run that waits for it fails. - Cancel of the pipeline cancels the waiting runs. After Approve, the runs of a pull request wait as usual. - A retry of a workflow also runs the workflows that it waits for, and the retried run waits for them again. A retry of a workflow that others wait for does not run those others again. ## Concurrency The `engine: cloudflare` runs of one repo, ref and workflow run one at a time. The ref is the branch, or the pull request for a pull request run. A new push or pull request run cancels the older waiting runs of its ref that are not more trusted. Manual runs and retries cancel no runs, and wait for their turn. Runs of other branches, pull requests or workflows run at the same time, up to 10 runs for each treadle deploy. treadle gives up on a run that cannot start within 3 days. `engine: buildkite` builds follow the concurrency settings of the Buildkite pipeline. ## Logs and retention For `engine: cloudflare`, treadle stores the log of each run for 30 days after the run ends, unless the deploy sets another `LOG_RETENTION_DAYS`. The log of each step stops after 2 MiB. For `engine: buildkite`, treadle reads the job logs from Buildkite when someone opens the log. It shows at most the last 2 MiB of each job. Buildkite keeps the full log. ## Webhooks The Tangled web app sets repo webhooks under Settings > Hooks with the `org.tangled.webhook.*` methods. treadle sends events as upstream spindles do, with the same headers, payloads and `X-Tangled-Signature-256` signature: - `push` for each updated branch or tag, also when the push skips CI or matches no workflow. A deleted ref sends nothing. - `pull_request:created` and `pull_request:resubmitted` for each new patch of an open pull request. - `pull_request:merged`, `pull_request:closed` and `pull_request:reopened` when the state of a pull request changes after treadle first saw it. Changes that come within about 10 seconds of each other can show as one change. A webhook can select `repository:renamed`, but treadle does not send it yet. treadle tries a delivery up to 3 times when the request fails or the endpoint answers with a 5xx status. It does not follow redirects. It keeps the last 50 deliveries of each webhook, with the first 10 KiB of each response. ## Differences from other spindles treadle reads the same workflow files as other Tangled spindles, with these differences: - treadle supports only `engine: buildkite` and `engine: cloudflare`. It skips workflows for other engines. - `dependencies` is a flat list of Debian packages. treadle skips an `engine: cloudflare` workflow whose `dependencies` is a map. - The `treadle` and `trusted` keys are treadle settings. - `runs_on` names a Cloudflare Containers instance type or a custom size, not executor labels. - A manual run ignores `when`. - The `engine: cloudflare` runs of one repo, ref and workflow run one at a time, and the newest waiting run replaces older waiting runs. - treadle serves only the repos of one owner. Pull requests from other people wait for Approve. - `engine: buildkite` builds get no secrets from treadle. - Webhooks do not send `repository:renamed`.