Was going to open as a PR, but as this is taking longer and I haven't officially get confirmed about this idea yet, I'm opening a proposal issue to show the high-level overview of ongoing spindle rewrite.

You can find the WIP implementation in sl/spindle-rewrite branch

Related Issue: #320 proposal: trigger pipeline from spindle (not from knot)
Related Discord Thread: message

(image) high level overview of new spindle architecture

So as stated from #320, it would be cool if we can let spindle to use whatever workflow definition spec they want. Only high level concept of "workflow" will be shared and the workflow definition can be completely different by spindle engines. They can use .woodpecker/*.yml, .github/workflows/*.yml or even non-yaml files for workflow definition.

Lexicon refactor to sh.tangled.ci.*#

"workflow" is not a sub-concept under "pipeline".

The CI terminology is pretty confusing and all CI services use different term. For example, from woodpecker's definition,

• Pipeline: A sequence of workflows that are executed on the code. Pipelines are triggered by events.
• Workflow: A sequence of steps and services that are executed as part of a pipeline. Workflows are represented by YAML files. Each workflow has its own isolated workspace, and often additional resources like a shared network (docker).

The term "workflow" here actually represents two different stuffs. If we divide them,

• workflow definition: a job definition usually represented in single YAML file
• workflow run: an actually triggered process for defined CI process.

Workflow definition is higher level concept than pipeline, while workflow run is lower level concept than pipeline. It's hard to define which is higher concept than another.

So it makes sense to put everything under sh.tangled.ci.* than sh.tangled.pipeline.*. Also, lexicon style guidelines recommends to avoid reusing group NSID for specific lexicon.

sh.tangled.ci.event#

This is not a record, but published, generalized type to define "trigger events".

It will hold all public metadata related to the trigger event. Basically same to current sh.tangled.pipeline#triggerMetadata. Just renaming it.

sh.tangled.ci.pipeline#

A immutable record stored in spindle. A pipeline record will hold:

• event that triggered the pipeline (sh.tangled.ci.event type object)
• list of AT-URI referencing sh.tangled.ci.workflow.run records

sh.tangled.ci.workflow.run#

A mutable record stored in spindle. Representing triggered "workflow run". A workflow run record will hold:

• workflow definition name (usually file name, but can be anything)
• current workflow run status (pending|running|failed|canceled|timeout|success)

sh.tangled.ci.triggerPipeline#

xrpc endpoint in spindle to create new sh.tangled.ci.pipeline record with "manual" type event.

sh.tangled.ci.cancelWorkflowRun | sh.tangled.ci.cancelPipeline#

xrpc endpoints in spindle to cancel specific workflow run or entire pipeline.

Spindle 'Adapters'#

Because spindle engines are becoming more generic (they can use their own way to define a workflow,) I suggest to rename current spindle 'engines' to 'adapters'.

An adapter will:

1. receive pipeline trigger event from spindle
2. compare with current workflow definition
3. and return triggered list of workflows (can be empty if none of them triggered)

And spindle will:

1. listen to relay/knotstream, create ci.event object and pass it to adapters
2. collect ci.workflow.run records returned from adapters
3. conform ci.pipeline record referencing triggered ci.workflow.run records
4. stream ci.workflow.run and ci.pipeline records to appview
5. update the ci.workflow.run record when run status changed.

In future, we might allow adapters running as separate binary communicating with spindle server via stdio or grpc, but for now, let's keep things simple and leave 'adapter' concept as just go interface (this is how current 'engine' works).

Example of spindle adapters#

1. tangled-nixery adapter using .tangled/*.yml files with engine: nixery for workflow definition and run workflows from nixery container.
2. woodpecker adapter using .woodpecker/*.yml files for workflow definition
3. XcodeCloud adapter using remotely configured workflows