Create content source plugin system #65

Overview #

I'd like to propose sort of a "plugin" approach to Sequoia's parsing logic to avoid the core team having to manage nearly infinite site setup options. My approach with this plan is to separate the pieces that Sequoia needs to do its job from how each site manages the content. I'd love to not have to think about the credentials piece, bluesky posting, the particulars of the standard.site lexicons, and other minutiae that Sequoia deals with. On the flip side I don't think Sequoia is realistically going to be able to know how I structured my /posts directory or how to pull the post date from the filename like Jekyll sites often do. This proposal is to try to define that boundary and how I could see it working.

Background #

Personally I had some trouble with onboarding as noted in my PR. I also saw Jekyll dates being a problem. And while Zola isn't something I'm familiar with it appears to have other patterns established. My own use case is a completely custom nextjs site where my content is structured and formatted in a way that makes sense to me and then I shove it into the right places in CICD to make the site work.

Current Publish Flow #

Before diving into the separation of Sequoia and the post fetching/parsing I wanted to share my "outline" of the current flow. This is a result of me just walking through the code in the publish command. Full disclosure: I didn't dig through other commands to see how this proposed plugin system might impact them.

1. Config (find and load config)
2. Credentials (load creds or select cred and optionally authenticate with that credential)
3. State (load existing state and optionally load state from the PDS)
4. Collect Posts (scan the configured directory for posts)
5. Determine Posts to Publish (skip drafts, resolve cover image, find post state, mark post with analysis result)
6. Bluesky Config (load and determine Bluesky plans)
7. Validate Posts (currently just cover image size check)
8. Sync Publication (ensure we have a standard.site publication)
9. Publish Posts (upload cover image, upsert document, update frontmatter, post to bluesky, save post-specific state)
10. Save State (write state back to disk for next time)

Content Source Plugins #

Steps 4, 5, 7, and 9 have dependencies on how the "source" of the content manages files, directories, and frontmatter. I think it would be entirely possible to define a contract for these interactions and call them something like "content source" plugins. Some initial ideas to hopefully help clarify the approach.

type CollectionResult = {
  post: BlogPost;
  action: "create" | "update " | "draft"
  reason: string
}
export async function collectPosts( contentDir ): Promise<CollectionResult[]> {}
export async function postsToPublishAreValid( postsToPublish: CollectionResult[] ): Promise<boolean> {}
export async function updateFrontmatterWithAtUri( postFilePath: string, atUri: string ): Promise<void> {}

Social Plugins #

This didn't occur to me until writing this issue up but the Bluesky piece could also be pulled out and put into a plugin. This could allow other platforms to be added either by the core team or individual users of Sequoia. I won't elaborate on this here as it'll muddy the waters but I think if there is interest in keeping the plugin approach this is another place the core of Sequoia could be kept lean.