Tangled Spindle built on Cloudflare Workers
treadle docs workflows.md
17 kB · 302 lines
Markdown

treadle workflows #

A workflow is a YAML file in .tangled/workflows/ of a repo. treadle reads every regular file in that directory whose name ends in .yml or .yaml. The file name is the name of the workflow. treadle skips a workflow whose engine it does not know.

Each event that matches one or more workflows makes one pipeline. The pipeline has one run for each matching workflow. The runs of a pipeline run in parallel, unless treadle.after orders them.

A file that treadle cannot parse makes a failed run, with the parse error as its error. treadle matches the run to events by the when of the file. When treadle cannot read the YAML or the when of the file, the run matches every event. A failed run of a pull request does not wait for Approve.

treadle reads at most 50 workflow files, each of at most 256 KiB. When a commit has more files or a larger file, a push or a manual run makes one failed run named .tangled/workflows, and a pull request gets no pipeline.

Engines #

The engine key selects where the workflow runs:

  • engine: buildkite starts a build of a Buildkite pipeline. Buildkite runs the steps of that pipeline. treadle ignores steps, environment, dependencies, secrets, cache and runs_on in the workflow file.
  • engine: cloudflare runs the steps of the workflow file in a Cloudflare container. treadle starts one container for each run.

treadle skips a workflow with any other engine, or with no engine.

Triggers #

when is a list of conditions. A workflow matches an event when one condition matches. Each field takes a string or a list of strings:

  • event: push, pull_request or manual.
  • branch: patterns for the branch of a push, or for the target branch of a pull request.
  • tag: patterns for the tag of a push.
  • types: the actions of a pull request that start a run. The default is opened, reopened and synchronize. treadle sends opened for the first patch of a pull request and synchronize for each later patch.

A workflow without when matches every push and every pull request.

when:
  - event: push
    branch: [main, "release/*"]
  - event: push
    tag: "v*"
  - event: pull_request
    branch: main
  - event: manual

Rules for each event:

  • A push to a branch matches a condition with event: push and a matching branch pattern. A push of a tag needs a matching tag pattern. A push of any other ref, or a push that deletes a ref, starts no run.
  • A push with the push option skip-ci or ci-skip starts no run.
  • A pull request matches a condition with event: pull_request, a branch pattern that matches the target branch, and a matching action.
  • A manual run ignores when. It runs every workflow of the commit, or only the workflows that it names and the workflows that they wait for with treadle.after.

Patterns #

  • * matches any characters except /.
  • ** matches across / when it is a whole path segment, for example ** or release/**.
  • ? matches one character except /.
  • [abc] matches one of the characters. [!abc] matches any other character.

Manual inputs #

A manual run can carry inputs, each with a name and a value. An input name has only letters, digits and underscores, and does not start with a digit. For engine: buildkite, the build gets each input as the variable TANGLED_INPUT_<NAME>, with the name in upper case. engine: cloudflare runs do not get the inputs.

Steps #

Only engine: cloudflare reads steps. It is a list that must not be empty. Each step needs a name and a command.

engine: cloudflare
steps:
  - name: Install
    command: npm ci
  - name: Test
    command: |
      npm test
      npm run lint

The steps run in order in /workspace, which holds the files of the commit without the Git history. Each command runs with bash -eo pipefail -c. The first step that fails stops the run, and the run fails.

The container uses the cloudflare/debian-trixie image and has internet access.

Environment #

Only engine: cloudflare reads environment. Set it for the whole workflow, for one step, or both. A step variable replaces a workflow variable with the same name.

environment:
  NODE_ENV: test
  PYTHON_VERSION: 3.10
steps:
  - name: Test
    command: npm test
    environment:
      DEBUG: "1"

Rules:

  • environment is a map. A name has letters, digits and underscores, and does not start with a digit.
  • A value is a string, a number or a boolean. treadle keeps a number as written, so 3.10 stays 3.10.
  • The map must not come from a YAML alias.

Each step also gets these variables:

  • CI=true, HOME=/root, and a standard PATH.
  • TREADLE_REPO, TREADLE_COMMIT, TREADLE_PIPELINE and TREADLE_WORKFLOW.
  • The secrets of the workflow, when the run is trusted. An injected secret holds a placeholder, and NODE_EXTRA_CA_CERTS is set.

An engine: buildkite build gets TREADLE_REPO, TREADLE_PIPELINE and TREADLE_WORKFLOW as build variables, and the same names as build meta-data with the prefix treadle:.

Dependencies #

Only engine: cloudflare reads dependencies. It is a flat list of Debian package names. A name can include a version, as in curl=8.14.1-2.

dependencies: [ca-certificates, git, python3]

treadle installs the packages with apt-get in a first step called Install dependencies. This step shows in the log, and it fails the run as other steps do.

Secrets #

Only engine: cloudflare reads secrets. It lists the repo secrets that the steps get as variables. An item is a secret name, or a map that injects the secret into HTTPS requests.

secrets:
  - NPM_TOKEN
  - name: CLOUDFLARE_API_TOKEN
    hosts: [api.cloudflare.com]
  - name: OTHER_KEY
    hosts: ["*.example.com"]
    headers: [X-Auth-Key]

Rules:

  • Only a trusted run gets secrets. An untrusted run gets none, and the variables are not set.
  • A trusted run fails when a secret that it names is not set.
  • treadle replaces each secret value in the log with ***. A log line with a secret loses its colors.
  • A name appears only once in the list.
  • The Tangled web app sets secrets with the org.tangled.secret.* methods. A secret name is a variable name of at most 50 characters. A value has 1 to 4096 bytes and no NUL character.

engine: buildkite builds get no secrets from treadle. Keep the secrets of a Buildkite build in Buildkite.

Injected secrets #

The value of an injected secret is not in the container. The variable holds a random placeholder, which is new for each run. treadle intercepts the HTTPS connections to the hosts of the secret. In a request to one of these hosts, treadle replaces the placeholder with the value in the headers of the secret. The URL and the body do not change.

  • hosts lists at most 64 lowercase hostnames on port 443, not IP addresses. *.example.com matches each subdomain of example.com, but not example.com.
  • headers lists the header names. The default is [Authorization]. In a Basic value, treadle also replaces the placeholder in the decoded user:password.
  • treadle replaces the placeholder only when the URL of the request has one of the hosts and port 443. A request to another host keeps the placeholder.
  • treadle does not follow redirects. The client in the container gets the redirect.
  • Node.js trusts the intercept CA through NODE_EXTRA_CA_CERTS. treadle also adds the CA to the system store when update-ca-certificates is installed, for example with dependencies: [ca-certificates]. Other clients may need their own setting, for example REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt for Python requests.
  • A secret that is not an HTTP header, such as an SSH key or a signing key, must be a plain name.

Injection stops only the theft of the stored value. For example, a dependency that sends the environment to another host gets only the placeholder, and the files of the container never hold the value. Injection does not stop the use of the secret: during a trusted run, each step can do on its hosts all that the secret allows, for example create a new token. treadle does not filter or mask the responses, so a run with injected secrets saves the trusted snapshot. Rules for hosts:

  • Never list a host that returns request headers in its response, because a step can then read the value.
  • Never list a host or glob where other people can create names, such as *.workers.dev or *.github.io.

Trust #

A trusted run of an engine: cloudflare workflow gets its secrets and can use the trusted cache. treadle decides trust as follows:

  • A push to a trusted branch is trusted. By default, only the default branch is trusted.
  • A manual run on a trusted branch is trusted only when its commit is the head of that branch.
  • A push of a tag is not trusted.
  • A pull request run is never trusted, whatever its branch.
  • A retry keeps the trust of the run that it repeats.

trusted.branch lists the trusted branches as patterns. treadle reads it only from the workflow file on the default branch, so a branch or a pull request cannot make itself trusted.

trusted:
  branch: [main, "release/*"]

Pull requests and Approve #

treadle finds new patches of pull requests on the repos that it serves. The runs of a pull request start at once when the repo owner opened the pull request and sent the patch. The runs of any other pull request wait for Approve in the Tangled web app. A retry of a pull request that did not get Approve also waits for Approve.

For each repo, treadle keeps at most 10 pull request pipelines from one sender that wait for Approve. A new patch from that sender deletes the oldest of them. treadle deletes a pipeline that waits for Approve for 14 days.

Cache and snapshots #

engine: cloudflare runs can start from a snapshot of an earlier container. Set cache: true to turn this on. It is off by default.

cache: true

After a successful trusted run at the head of the default branch, treadle removes /workspace, its own files and /tmp, and saves a snapshot of the container. The snapshot keeps the whole root file system, so it also keeps every file that a step or a dependency changed outside those directories. This shows as a last step called Save cache. A failed snapshot does not fail the run.

treadle keeps up to two snapshots for each workflow:

  • The trusted snapshot comes from a run that had secrets, or that started from the trusted snapshot. Only trusted runs start from it.
  • The public snapshot comes from any other trusted run. Every run can start from it.

A newer public snapshot replaces an older trusted snapshot, so trusted runs then start from the public snapshot. When a snapshot does not start, the run starts again from the image.

A run starts from a snapshot only when its workflow file is the same as the file of the run that saved the snapshot. After a change to the workflow file, runs start from the image again. After a successful trusted run with cache: false at the head of the default branch, treadle discards the saved snapshots of the workflow.

Instance types and timeouts #

runs_on sets the container size. The default is standard-1. Give the name of an instance type, such as lite, basic or standard-2, or a custom size with vcpu, memory_mib and disk_mb. Cloudflare Containers checks the values when the container starts, and the run fails if they are not valid.

treadle.cloudflare.timeout sets the time limit of a run. The default is 1h, and the maximum is 6h. Write it with hours, minutes and seconds, as in 1h30m, 45m or 90s. A run that is still running at the limit ends with the status timeout.

timeout is the only key of treadle.cloudflare. treadle rejects a workflow with any other key there.

runs_on: standard-2
treadle:
  cloudflare:
    timeout: 20m
runs_on:
  vcpu: 2
  memory_mib: 8192
  disk_mb: 16000

Buildkite settings #

treadle.buildkite sets the Buildkite pipeline of an engine: buildkite workflow:

  • pipeline: the pipeline slug. The default is the repo name in lower case, with each run of other characters replaced by -, and with no - at the start or end. treadle skips the workflow when this default is empty.
  • org: the Buildkite organization. The default is the organization of the treadle deploy.
  • clean_checkout: true or false, sent to Buildkite as the clean checkout setting of the build.

A pull request run takes pipeline and org from the file of the same name on the default branch, so a pull request cannot choose the pipeline that builds it. When the default branch has no such file, the run uses the defaults.

engine: buildkite
when:
  - event: push
    branch: main
treadle:
  buildkite:
    pipeline: my-app
    clean_checkout: true

The build gets the commit, the branch, the commit message and author, and for a pull request the target branch. Buildkite branch filters of the pipeline do not apply.

Order #

treadle.after lists the workflows of the same pipeline that must succeed before the run starts. A name can leave out .yml or .yaml, so check names check.yml. It works for both engines. A waiting run has the status pending. It has no container and no Buildkite build, and it does not count toward the limit of 10 runs.

engine: cloudflare
secrets: [CLOUDFLARE_API_TOKEN]
treadle:
  after: [check.yml]

Rules:

  • The run starts when every workflow that it names has the status success.
  • When one of them fails, times out or is cancelled, or its file is not valid, the run is cancelled with an error that names it.
  • A name that is not a workflow of the pipeline, or a list that makes a cycle, fails the run at once. A workflow that does not match the event is not in the pipeline, so a run that waits for it fails.
  • Cancel of the pipeline cancels the waiting runs. After Approve, the runs of a pull request wait as usual.
  • A retry of a workflow also runs the workflows that it waits for, and the retried run waits for them again. A retry of a workflow that others wait for does not run those others again.

Concurrency #

The engine: cloudflare runs of one repo, ref and workflow run one at a time. The ref is the branch, or the pull request for a pull request run. A new push or pull request run cancels the older waiting runs of its ref that are not more trusted. Manual runs and retries cancel no runs, and wait for their turn. Runs of other branches, pull requests or workflows run at the same time, up to 10 runs for each treadle deploy.

treadle gives up on a run that cannot start within 3 days.

engine: buildkite builds follow the concurrency settings of the Buildkite pipeline.

Logs and retention #

For engine: cloudflare, treadle stores the log of each run for 30 days after the run ends, unless the deploy sets another LOG_RETENTION_DAYS. The log of each step stops after 2 MiB.

For engine: buildkite, treadle reads the job logs from Buildkite when someone opens the log. It shows at most the last 2 MiB of each job. Buildkite keeps the full log.

Webhooks #

The Tangled web app sets repo webhooks under Settings > Hooks with the org.tangled.webhook.* methods. treadle sends events as upstream spindles do, with the same headers, payloads and X-Tangled-Signature-256 signature:

  • push for each updated branch or tag, also when the push skips CI or matches no workflow. A deleted ref sends nothing.
  • pull_request:created and pull_request:resubmitted for each new patch of an open pull request.
  • pull_request:merged, pull_request:closed and pull_request:reopened when the state of a pull request changes after treadle first saw it. Changes that come within about 10 seconds of each other can show as one change.

A webhook can select repository:renamed, but treadle does not send it yet.

treadle tries a delivery up to 3 times when the request fails or the endpoint answers with a 5xx status. It does not follow redirects. It keeps the last 50 deliveries of each webhook, with the first 10 KiB of each response.

Differences from other spindles #

treadle reads the same workflow files as other Tangled spindles, with these differences:

  • treadle supports only engine: buildkite and engine: cloudflare. It skips workflows for other engines.
  • dependencies is a flat list of Debian packages. treadle skips an engine: cloudflare workflow whose dependencies is a map.
  • The treadle and trusted keys are treadle settings.
  • runs_on names a Cloudflare Containers instance type or a custom size, not executor labels.
  • A manual run ignores when.
  • The engine: cloudflare runs of one repo, ref and workflow run one at a time, and the newest waiting run replaces older waiting runs.
  • treadle serves only the repos of one owner. Pull requests from other people wait for Approve.
  • engine: buildkite builds get no secrets from treadle.
  • Webhooks do not send repository:renamed.