Syncing is pretty complicated task. We should consider backfilling, event orders and all sort of stuffs. If it's unnecessary, it's better to avoid it. As I mentioned from my Knot-stored COB proposal, Spindle is more closer to BlueSky Feed Generator than just Data Server.
- We don't globally index pipelines.
- We are fine with pipelines tied to the spindle instance.
"Owning pipelines" or "Migrating pipelines between spindles" doesn't make much sense. You should own the automation part, not result part.
Thus, we don't need sync for pipelines and then it is way simpler to just fetch pipelines on-demand like Bluesky Feeds. AppView doesn't have to know the real-time status of each pipelines and even if we do need it in future (e.g. AppView-side notification,) syncing will be completely optional and won't be a core requirement for running the infra.
Implementation #
spindle will expose two APIs to fetch pipelines:
org.tangled.ci.listPipelines(repo DID, commits []OID)- receive repo DID & commit OIDs and return list of pipeline statuses associated to each commits.org.tangled.ci.subscribePipelineLogs(pipeline string)- receive pipeline ID and stream real-time logs. similar to current/logsendpoint.
On page render, AppView will lazy-load pipeline statuses. In this style, it becomes way easier to periodically update each pipeline statuses in real-time.
tl;dr
- pipelines are similar to feedgen, so we don't have to sync pipelines
- spindle event stream will be optional and will be removed for simplicity
Issues like #616 can benefit from this since appview won't fail silently when websocket connection fails.
Instead, it can show clear error messages when we actually need the event stream; the log page.