drornir.dev
my monorepo for atproto based applications
1# ATProto Feed Generator
2
3This is a starter kit for creating ATProto Feed Generators. It's not feature complete, but should give you a good starting ground off of which to build and deploy a feed.
4
5## Overview
6
7Feed Generators are services that provide custom algorithms to users through the AT Protocol.
8
9They work very simply: the server receives a request from a user's server and returns a list of [post URIs](https://atproto.com/specs/at-uri-scheme) with some optional metadata attached. Those posts are then hydrated into full views by the requesting server and sent back to the client. This route is described in the [`app.bsky.feed.getFeedSkeleton` lexicon](https://docs.bsky.app/docs/api/app-bsky-feed-get-feed-skeleton).
10
11A Feed Generator service can host one or more algorithms. The service itself is identified by DID, while each algorithm that it hosts is declared by a record in the repo of the account that created it. For instance, feeds offered by Bluesky will likely be declared in `@bsky.app`'s repo. Therefore, a given algorithm is identified by the at-uri of the declaration record. This declaration record includes a pointer to the service's DID along with some profile information for the feed.
12
13The general flow of providing a custom algorithm to a user is as follows:
14- A user requests a feed from their server (PDS) using the at-uri of the declared feed
15- The PDS resolves the at-uri and finds the DID doc of the Feed Generator
16- The PDS sends a `getFeedSkeleton` request to the service endpoint declared in the Feed Generator's DID doc
17 - This request is authenticated by a JWT signed by the user's repo signing key
18- The Feed Generator returns a skeleton of the feed to the user's PDS
19- The PDS hydrates the feed (user info, post contents, aggregates, etc.)
20 - In the future, the PDS will hydrate the feed with the help of an App View, but for now, the PDS handles hydration itself
21- The PDS returns the hydrated feed to the user
22
23For users, this should feel like visiting a page in the app. Once they subscribe to a custom algorithm, it will appear in their home interface as one of their available feeds.
24
25## Getting Started
26
27We've set up this simple server with SQLite to store and query data. Feel free to switch this out for whichever database you prefer.
28
29Next, you will need to do two things:
30
311. Implement indexing logic in `src/subscription.ts`.
32
33 This will subscribe to the repo subscription stream on startup, parse events and index them according to your provided logic.
34
352. Implement feed generation logic in `src/algos`
36
37 For inspiration, we've provided a very simple feed algorithm (`whats-alf`) that returns all posts related to the titular character of the TV show ALF.
38
39 You can either edit it or add another algorithm alongside it. The types are in place, and you will just need to return something that satisfies the `SkeletonFeedPost[]` type.
40
41We've taken care of setting this server up with a did:web. However, you're free to switch this out for did:plc if you like - you may want to if you expect this Feed Generator to be long-standing and possibly migrating domains.
42
43### Deploying your feed
44Your feed will need to be accessible at the value supplied to the `FEEDGEN_HOSTNAME` environment variable.
45
46The service must be set up to respond to HTTPS queries over port 443.
47
48### Publishing your feed
49
50To publish your feed, go to the script at `scripts/publishFeedGen.ts` and fill in the variables at the top. Examples are included, and some are optional. To publish your feed generator, simply run `yarn publishFeed`.
51
52To update your feed's display data (name, avatar, description, etc.), just update the relevant variables and re-run the script.
53
54After successfully running the script, you should be able to see your feed from within the app, as well as share it by embedding a link in a post (similar to a quote post).
55
56## Running the Server
57
58Install dependencies with `yarn` and then run the server with `yarn start`. This will start the server on port 3000, or what's defined in `.env`. You can then watch the firehose output in the console and access the output of the default custom ALF feed at [http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf](http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf).
59
60## Some Details
61
62### Skeleton Metadata
63
64The skeleton that a Feed Generator puts together is, in its simplest form, a list of post URIs.
65
66```ts
67[
68 {post: 'at://did:example:1234/app.bsky.feed.post/1'},
69 {post: 'at://did:example:1234/app.bsky.feed.post/2'},
70 {post: 'at://did:example:1234/app.bsky.feed.post/3'}
71]
72```
73
74However, we include an additional location to attach some context. Here is the full schema:
75
76```ts
77type SkeletonItem = {
78 post: string // post URI
79
80 // optional reason for inclusion in the feed
81 // (generally to be displayed in client)
82 reason?: Reason
83}
84
85// for now, the only defined reason is a repost, but this is open to extension
86type Reason = ReasonRepost
87
88type ReasonRepost = {
89 $type: 'app.bsky.feed.defs#skeletonReasonRepost'
90 repost: string // repost URI
91}
92```
93
94This metadata serves two purposes:
95
961. To aid the PDS in hydrating all relevant post information
972. To give a cue to the client in terms of context to display when rendering a post
98
99### Authentication
100
101If you are creating a generic feed that does not differ for different users, you do not need to check auth. But if a user's state (such as follows or likes) is taken into account, we _strongly_ encourage you to validate their auth token.
102
103Users are authenticated with a simple JWT signed by the user's repo signing key.
104
105This JWT header/payload takes the format:
106```ts
107const header = {
108 type: "JWT",
109 alg: "ES256K" // (key algorithm) - in this case secp256k1
110}
111const payload = {
112 iss: "did:example:alice", // (issuer) the requesting user's DID
113 aud: "did:example:feedGenerator", // (audience) the DID of the Feed Generator
114 exp: 1683643619 // (expiration) unix timestamp in seconds
115}
116```
117
118We provide utilities for verifying user JWTs in the `@atproto/xrpc-server` package, and you can see them in action in `src/auth.ts`.
119
120### Pagination
121You'll notice that the `getFeedSkeleton` method returns a `cursor` in its response and takes a `cursor` param as input.
122
123This cursor is treated as an opaque value and fully at the Feed Generator's discretion. It is simply passed through the PDS directly to and from the client.
124
125We strongly encourage that the cursor be _unique per feed item_ to prevent unexpected behavior in pagination.
126
127We recommend, for instance, a compound cursor with a timestamp + a CID:
128`1683654690921::bafyreia3tbsfxe3cc75xrxyyn6qc42oupi73fxiox76prlyi5bpx7hr72u`
129
130## Suggestions for Implementation
131
132How a feed generator fulfills the `getFeedSkeleton` request is completely at their discretion. At the simplest end, a Feed Generator could supply a "feed" that only contains some hardcoded posts.
133
134For most use cases, we recommend subscribing to the firehose at `com.atproto.sync.subscribeRepos`. This websocket will send you every record that is published on the network. Since Feed Generators do not need to provide hydrated posts, you can index as much or as little of the firehose as necessary.
135
136Depending on your algorithm, you likely do not need to keep posts around for long. Unless your algorithm is intended to provide "posts you missed" or something similar, you can likely garbage collect any data that is older than 48 hours.
137
138Some examples:
139
140### Reimplementing What's Hot
141To reimplement "What's Hot", you may subscribe to the firehose and filter for all posts and likes (ignoring profiles/reposts/follows/etc.). You would keep a running tally of likes per post and when a PDS requests a feed, you would send the most recent posts that pass some threshold of likes.
142
143### A Community Feed
144You might create a feed for a given community by compiling a list of DIDs within that community and filtering the firehose for all posts from users within that list.
145
146### A Topical Feed
147To implement a topical feed, you might filter the algorithm for posts and pass the post text through some filtering mechanism (an LLM, a keyword matcher, etc.) that filters for the topic of your choice.
148
149## Community Feed Generator Templates
150
151- [Python](https://github.com/MarshalX/bluesky-feed-generator) - [@MarshalX](https://github.com/MarshalX)
152- [Ruby](https://github.com/mackuba/bluesky-feeds-rb) - [@mackuba](https://github.com/mackuba)