comet.sh
Compare changes
Choose any two refs to compare.
+2
-18
.gitignore
+2
-18
.gitignore
···
1
-
# The directory Mix will write compiled artifacts to.
2
1
/_build/
3
-
4
-
# If you run "mix test --cover", coverage assets end up here.
5
2
/cover/
6
-
7
-
# The directory Mix downloads your dependencies sources to.
8
3
/deps/
9
-
10
-
# Where third-party dependencies like ExDoc output generated docs.
11
4
/doc/
12
-
13
-
# If the VM crashes, it generates a dump, let's ignore it too.
14
5
erl_crash.dump
15
-
16
-
# Also ignore archive artifacts (built via "mix archive.build").
17
6
*.ez
18
-
19
-
# Ignore package tarball (built via "mix hex.build").
20
7
drinkup-*.tar
21
-
22
-
# Temporary files, for example, from tests.
23
8
/tmp/
24
-
25
-
# Nix
26
9
.envrc
27
10
.direnv
28
-
result
11
+
result
12
+
priv/dets/
+40
AGENTS.md
+40
AGENTS.md
···
1
+
# Agent Guidelines for Drinkup
2
+
3
+
## Commands
4
+
5
+
- **Test**: `mix test` (all), `mix test test/path/to/file_test.exs` (single file), `mix test test/path/to/file_test.exs:42` (single test at line)
6
+
- **Format**: `mix format` (auto-formats all code)
7
+
- **Lint**: `mix credo` (static analysis), `mix credo --strict` (strict mode)
8
+
- **Compile**: `mix compile`
9
+
- **Docs**: `mix docs`
10
+
- **Type Check**: `mix dialyzer` (if configured)
11
+
12
+
## Code Style
13
+
14
+
- **Imports**: Use `alias` for modules (e.g., `alias Drinkup.Firehose.{Event, Options}`), `require` for macros (e.g., `require Logger`)
15
+
- **Formatting**: Elixir 1.18+, auto-formatted via `.formatter.exs` with `import_deps: [:typedstruct]`
16
+
- **Naming**: snake_case for functions/variables, PascalCase for modules, `:lowercase_atoms` for atoms, `@behaviour` (not `@behavior`)
17
+
- **Types**: Use `@type` and `@spec` for all functions; use TypedStruct for structs with `enforce: true` for required fields
18
+
- **Moduledocs**: Public modules need `@moduledoc`, public functions need `@doc` with examples
19
+
- **Error Handling**: Return `{:ok, result}` or `{:error, reason}` tuples; use `with` for chaining operations; log errors with `Logger.error("#{Exception.format(:error, e, __STACKTRACE__)}")`
20
+
- **Pattern Matching**: Prefer pattern matching in function heads over conditionals; use guard clauses when appropriate
21
+
- **OTP**: Use `child_spec/1` for custom supervisor specs; `:gen_statem` for state machines; `Task.Supervisor` for concurrent tasks; Registry for named lookups
22
+
- **Tests**: Use ExUnit with `use ExUnit.Case`; use `doctest Module` for documentation examples
23
+
- **Dependencies**: Core deps include gun (WebSocket), car (CAR format), cbor (encoding), TypedStruct (typed structs), Credo (linting)
24
+
25
+
## Project Structure
26
+
27
+
- **Namespace**: All firehose functionality under `Drinkup.Firehose.*`
28
+
- `Drinkup.Firehose` - Main supervisor
29
+
- `Drinkup.Firehose.Consumer` - Behaviour for handling all events
30
+
- `Drinkup.Firehose.RecordConsumer` - Macro for handling commit record events with filtering
31
+
- `Drinkup.Firehose.Event` - Event types (`Commit`, `Sync`, `Identity`, `Account`, `Info`)
32
+
- `Drinkup.Firehose.Socket` - `:gen_statem` WebSocket connection manager
33
+
- **Consumer Pattern**: Implement `@behaviour Drinkup.Firehose.Consumer` with `handle_event/1`
34
+
- **RecordConsumer Pattern**: `use Drinkup.Firehose.RecordConsumer, collections: [~r/app\.bsky\.graph\..+/, "app.bsky.feed.post"]` with `handle_create/1`, `handle_update/1`, `handle_delete/1` overrides
35
+
36
+
## Important Notes
37
+
38
+
- **Update CHANGELOG.md** when adding features, changes, or fixes under `## [Unreleased]` with appropriate sections (`Added`, `Changed`, `Fixed`, `Deprecated`, `Removed`, `Security`)
39
+
- **WebSocket States**: Socket uses `:disconnected` → `:connecting_http` → `:connecting_ws` → `:connected` flow
40
+
- **Sequence Tracking**: Use `Event.valid_seq?/2` to validate sequence numbers from firehose
+22
-2
CHANGELOG.md
+22
-2
CHANGELOG.md
···
6
6
and this project adheres to
7
7
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
8
8
9
+
## [Unreleased]
10
+
11
+
### Breaking Changes
12
+
13
+
- Existing behaviour moved to `Drinkup.Firehose` namespace, to make way for
14
+
alternate sync systems.
15
+
16
+
### Added
17
+
18
+
- Support for the
19
+
[Tap](https://github.com/bluesky-social/indigo/blob/main/cmd/tap/README.md)
20
+
sync and backfill utility service, via `Drinkup.Tap`.
21
+
- Support for [Jetstream](https://github.com/bluesky-social/jetstream), a
22
+
simplified JSON event stream for ATProto, via `Drinkup.Jetstream`.
23
+
24
+
### Changed
25
+
26
+
- Refactor core connection logic for websockets into `Drinkup.Socket` to make it
27
+
easy to use across multiple different services.
28
+
9
29
## [0.1.0] - 2025-05-26
10
30
11
31
Initial release.
12
32
13
-
[unreleased]: https://github.com/cometsh/elixir-car/compare/v0.1.0...HEAD
14
-
[0.1.0]: https://github.com/cometsh/elixir-car/releases/tag/v0.1.0
33
+
[unreleased]: https://github.com/cometsh/drinkup/compare/v0.1.0...HEAD
34
+
[0.1.0]: https://github.com/cometsh/drinkup/releases/tag/v0.1.0
+60
-58
README.md
+60
-58
README.md
···
1
1
# Drinkup
2
2
3
-
An Elixir library for listening to events from an ATProtocol relay
4
-
(firehose/`com.atproto.sync.subscribeRepos`). Eventually aiming to support any
5
-
ATProtocol subscription.
3
+
An Elixir library for consuming various AT Protocol sync services.
6
4
7
-
## TODO
5
+
Drinkup provides a unified interface for connecting to various AT Protocol data
6
+
streams, handling reconnection logic, sequence tracking, and event dispatch.
7
+
Choose the sync service that fits your needs:
8
8
9
-
- Support for different subscriptions other than
10
-
`com.atproto.sync.subscribeRepo'.
11
-
- Validation (signatures, making sure to only track handle active accounts,
12
-
etc.) (see
13
-
[Firehose Validation Best Practices](https://atproto.com/specs/sync#firehose-validation-best-practices))
14
-
- Look into backfilling? See if there's better ways to do it.
15
-
- Built-in solutions for tracking resumption? (probably a pluggable solution to
16
-
allow for different things like Mnesia, Postgres, etc.)
17
-
- Testing of multi-node/distribution.
18
-
- Tests
19
-
- Documentation
9
+
- **Firehose** - Raw event stream from the full AT Protocol network.
10
+
- **Jetstream** - Lightweight, cherry-picked event stream with filtering by
11
+
record collections and DIDs.
12
+
- **Tap** - Managed backfill and indexing solution.
20
13
21
14
## Installation
22
15
23
-
Add `drinkup` to your `mix.exs`.
16
+
Add `drinkup` to your `mix.exs`:
24
17
25
18
```elixir
26
19
def deps do
···
32
25
33
26
Documentation can be found on HexDocs at https://hexdocs.pm/drinkup.
34
27
35
-
## Example Usage
28
+
## Quick Start
36
29
37
-
First, create a module implementing the `Drinkup.Consumer` behaviour (only
38
-
requires a `handle_event/1` function):
30
+
### Firehose
39
31
40
32
```elixir
41
-
defmodule ExampleConsumer do
42
-
@behaviour Drinkup.Consumer
33
+
defmodule MyApp.FirehoseConsumer do
34
+
@behaviour Drinkup.Firehose.Consumer
43
35
44
-
def handle_event(%Drinkup.Event.Commit{} = event) do
45
-
IO.inspect(event, label: "Got commit event")
36
+
def handle_event(%Drinkup.Firehose.Event.Commit{} = event) do
37
+
IO.inspect(event, label: "Commit")
46
38
end
47
39
48
40
def handle_event(_), do: :noop
49
41
end
42
+
43
+
# In your supervision tree:
44
+
children = [{Drinkup.Firehose, %{consumer: MyApp.FirehoseConsumer}}]
50
45
```
51
46
52
-
Then add Drinkup and your consumer to your application's supervision tree:
47
+
### Jetstream
53
48
54
49
```elixir
55
-
defmodule MyApp.Application do
56
-
use Application
50
+
defmodule MyApp.JetstreamConsumer do
51
+
@behaviour Drinkup.Jetstream.Consumer
57
52
58
-
def start(_type, _args) do
59
-
children = [{Drinkup, %{consumer: ExampleConsumer}}]
60
-
Supervisor.start_link(children, strategy: :one_for_one)
53
+
def handle_event(%Drinkup.Jetstream.Event.Commit{} = event) do
54
+
IO.inspect(event, label: "Commit")
61
55
end
56
+
57
+
def handle_event(_), do: :noop
62
58
end
63
-
```
64
59
65
-
You should then be able to start your application and start seeing
66
-
`Got commit event: ...` in the terminal.
67
-
68
-
### Record Consumer
60
+
# In your supervision tree:
61
+
children = [
62
+
{Drinkup.Jetstream, %{
63
+
consumer: MyApp.JetstreamConsumer,
64
+
wanted_collections: ["app.bsky.feed.post"]
65
+
}}
66
+
]
67
+
```
69
68
70
-
One of the main reasons for listening to an ATProto relay is to synchronise a
71
-
database with records. As a result, Drinkup provides a light extension around a
72
-
basic consumer, the `RecordConsumer`, which only listens to commit events, and
73
-
transforms them into a slightly nicer structure to work around, calling your
74
-
`handle_create/1`, `handle_update/1`, and `handle_delete/1` functions for each
75
-
record it comes across. It also allows for filtering of specific types of
76
-
records either by full name or with a
77
-
[Regex](https://hexdocs.pm/elixir/1.18.4/Regex.html) match.
69
+
### Tap
78
70
79
71
```elixir
80
-
defmodule ExampleRecordConsumer do
81
-
# Will respond to any events either `app.bsky.feed.post` records, or anything under `app.bsky.graph`.
82
-
use Drinkup.RecordConsumer, collections: [~r/app\.bsky\.graph\..+/, "app.bsky.feed.post"]
83
-
alias Drinkup.RecordConsumer.Record
72
+
defmodule MyApp.TapConsumer do
73
+
@behaviour Drinkup.Tap.Consumer
84
74
85
-
def handle_create(%Record{type: "app.bsky.feed.post"} = record) do
86
-
IO.inspect(record, label: "Bluesky post created")
75
+
def handle_event(%Drinkup.Tap.Event.Record{} = event) do
76
+
IO.inspect(event, label: "Record")
87
77
end
88
78
89
-
def handle_create(%Record{type: "app.bsky.graph" <> _} = record) do
90
-
IO.inspect(record, label: "Bluesky graph updated")
91
-
end
79
+
def handle_event(_), do: :noop
80
+
end
92
81
93
-
def handle_update(record) do
94
-
# ...
95
-
end
82
+
# In your supervision tree:
83
+
children = [
84
+
{Drinkup.Tap, %{
85
+
consumer: MyApp.TapConsumer,
86
+
host: "http://localhost:2480"
87
+
}}
88
+
]
96
89
97
-
def handle_delete(record) do
98
-
# ...
99
-
end
100
-
end
90
+
# Track specific repos:
91
+
Drinkup.Tap.add_repos(Drinkup.Tap, ["did:plc:abc123"])
101
92
```
93
+
94
+
See [the examples](./examples) for some more complete samples.
95
+
96
+
## TODO
97
+
98
+
- Validation for Firehose events (signatures, active account tracking) — see
99
+
[Firehose Validation Best Practices](https://atproto.com/specs/sync#firehose-validation-best-practices)
100
+
- Pluggable cursor persistence (Mnesia, Postgres, etc.)
101
+
- Multi-node/distribution testing
102
+
- More comprehensive test coverage
103
+
- Additional documentation
102
104
103
105
## Special thanks
104
106
+14
compose.yml
+14
compose.yml
···
1
+
services:
2
+
tap:
3
+
image: "ghcr.io/bluesky-social/indigo/tap"
4
+
restart: "unless-stopped"
5
+
ports:
6
+
- "127.0.0.1:2480:2480"
7
+
volumes:
8
+
- "tap_data:/data"
9
+
environment:
10
+
TAP_SIGNAL_COLLECTION: "sh.weaver.actor.profile"
11
+
TAP_COLLECTION_FILTERS: "sh.weaver.*"
12
+
13
+
volumes:
14
+
tap_data:
-26
examples/basic_consumer.ex
-26
examples/basic_consumer.ex
···
1
-
defmodule BasicConsumer do
2
-
@behaviour Drinkup.Consumer
3
-
4
-
def handle_event(%Drinkup.Event.Commit{} = event) do
5
-
IO.inspect(event, label: "Got commit event")
6
-
end
7
-
8
-
def handle_event(_), do: :noop
9
-
end
10
-
11
-
defmodule ExampleSupervisor do
12
-
use Supervisor
13
-
14
-
def start_link(arg \\ []) do
15
-
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
16
-
end
17
-
18
-
@impl true
19
-
def init(_) do
20
-
children = [
21
-
{Drinkup, %{consumer: BasicConsumer}}
22
-
]
23
-
24
-
Supervisor.init(children, strategy: :one_for_one)
25
-
end
26
-
end
+26
examples/firehose/basic_consumer.ex
+26
examples/firehose/basic_consumer.ex
···
1
+
defmodule BasicConsumer do
2
+
@behaviour Drinkup.Firehose.Consumer
3
+
4
+
def handle_event(%Drinkup.Firehose.Event.Commit{} = event) do
5
+
IO.inspect(event, label: "Got commit event")
6
+
end
7
+
8
+
def handle_event(_), do: :noop
9
+
end
10
+
11
+
defmodule ExampleSupervisor do
12
+
use Supervisor
13
+
14
+
def start_link(arg \\ []) do
15
+
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
16
+
end
17
+
18
+
@impl true
19
+
def init(_) do
20
+
children = [
21
+
{Drinkup.Firehose, %{consumer: BasicConsumer}}
22
+
]
23
+
24
+
Supervisor.init(children, strategy: :one_for_one)
25
+
end
26
+
end
+35
examples/firehose/multiple_consumers.ex
+35
examples/firehose/multiple_consumers.ex
···
1
+
defmodule PostDeleteConsumer do
2
+
use Drinkup.Firehose.RecordConsumer, collections: ["app.bsky.feed.post"]
3
+
4
+
def handle_delete(record) do
5
+
IO.inspect(record, label: "update")
6
+
end
7
+
end
8
+
9
+
defmodule IdentityConsumer do
10
+
@behaviour Drinkup.Firehose.Consumer
11
+
12
+
def handle_event(%Drinkup.Firehose.Event.Identity{} = event) do
13
+
IO.inspect(event, label: "identity event")
14
+
end
15
+
16
+
def handle_event(_), do: :noop
17
+
end
18
+
19
+
defmodule ExampleSupervisor do
20
+
use Supervisor
21
+
22
+
def start_link(arg \\ []) do
23
+
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
24
+
end
25
+
26
+
@impl true
27
+
def init(_) do
28
+
children = [
29
+
{Drinkup.Firehose, %{consumer: PostDeleteConsumer}},
30
+
{Drinkup.Firehose, %{consumer: IdentityConsumer, name: :identities}}
31
+
]
32
+
33
+
Supervisor.init(children, strategy: :one_for_one)
34
+
end
35
+
end
+33
examples/firehose/record_consumer.ex
+33
examples/firehose/record_consumer.ex
···
1
+
defmodule ExampleRecordConsumer do
2
+
use Drinkup.Firehose.RecordConsumer,
3
+
collections: [~r/app\.bsky\.graph\..+/, "app.bsky.feed.post"]
4
+
5
+
def handle_create(record) do
6
+
IO.inspect(record, label: "create")
7
+
end
8
+
9
+
def handle_update(record) do
10
+
IO.inspect(record, label: "update")
11
+
end
12
+
13
+
def handle_delete(record) do
14
+
IO.inspect(record, label: "delete")
15
+
end
16
+
end
17
+
18
+
defmodule ExampleSupervisor do
19
+
use Supervisor
20
+
21
+
def start_link(arg \\ []) do
22
+
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
23
+
end
24
+
25
+
@impl true
26
+
def init(_) do
27
+
children = [
28
+
{Drinkup.Firehose, %{consumer: ExampleRecordConsumer}}
29
+
]
30
+
31
+
Supervisor.init(children, strategy: :one_for_one)
32
+
end
33
+
end
+118
examples/jetstream/jetstream_consumer.ex
+118
examples/jetstream/jetstream_consumer.ex
···
1
+
defmodule JetstreamConsumer do
2
+
@moduledoc """
3
+
Example Jetstream consumer implementation.
4
+
5
+
This consumer demonstrates handling different types of Jetstream events:
6
+
- Commit events (create, update, delete operations)
7
+
- Identity events (handle changes, etc.)
8
+
- Account events (status changes)
9
+
"""
10
+
11
+
@behaviour Drinkup.Jetstream.Consumer
12
+
13
+
def handle_event(%Drinkup.Jetstream.Event.Commit{operation: :create} = event) do
14
+
IO.inspect(event, label: "New record created")
15
+
:ok
16
+
end
17
+
18
+
def handle_event(%Drinkup.Jetstream.Event.Commit{operation: :update} = event) do
19
+
IO.inspect(event, label: "Record updated")
20
+
:ok
21
+
end
22
+
23
+
def handle_event(%Drinkup.Jetstream.Event.Commit{operation: :delete} = event) do
24
+
IO.inspect(event, label: "Record deleted")
25
+
:ok
26
+
end
27
+
28
+
def handle_event(%Drinkup.Jetstream.Event.Identity{} = event) do
29
+
IO.inspect(event, label: "Identity updated")
30
+
:ok
31
+
end
32
+
33
+
def handle_event(%Drinkup.Jetstream.Event.Account{active: false} = event) do
34
+
IO.inspect(event, label: "Account inactive")
35
+
:ok
36
+
end
37
+
38
+
def handle_event(%Drinkup.Jetstream.Event.Account{active: true} = event) do
39
+
IO.inspect(event, label: "Account active")
40
+
:ok
41
+
end
42
+
43
+
def handle_event(event) do
44
+
IO.inspect(event, label: "Unknown event")
45
+
:ok
46
+
end
47
+
end
48
+
49
+
defmodule ExampleJetstreamSupervisor do
50
+
@moduledoc """
51
+
Example supervisor that starts a Jetstream connection.
52
+
53
+
## Usage
54
+
55
+
# Start the supervisor
56
+
{:ok, pid} = ExampleJetstreamSupervisor.start_link()
57
+
58
+
# Update filters dynamically
59
+
Drinkup.Jetstream.update_options(MyJetstream, %{
60
+
wanted_collections: ["app.bsky.feed.post", "app.bsky.feed.like"]
61
+
})
62
+
63
+
# Stop the supervisor
64
+
Supervisor.stop(pid)
65
+
"""
66
+
67
+
use Supervisor
68
+
69
+
def start_link(arg \\ []) do
70
+
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
71
+
end
72
+
73
+
@impl true
74
+
def init(_) do
75
+
children = [
76
+
# Connect to public Jetstream instance and filter for posts and likes
77
+
{Drinkup.Jetstream,
78
+
%{
79
+
consumer: JetstreamConsumer,
80
+
name: MyJetstream,
81
+
wanted_collections: ["app.bsky.feed.post", "app.bsky.feed.like"]
82
+
}}
83
+
]
84
+
85
+
Supervisor.init(children, strategy: :one_for_one)
86
+
end
87
+
end
88
+
89
+
# Example: Filter for all graph operations (follows, blocks, etc.)
90
+
defmodule GraphEventsConsumer do
91
+
@behaviour Drinkup.Jetstream.Consumer
92
+
93
+
def handle_event(%Drinkup.Jetstream.Event.Commit{collection: "app.bsky.graph." <> _} = event) do
94
+
IO.puts("Graph event: #{event.collection} - #{event.operation}")
95
+
:ok
96
+
end
97
+
98
+
def handle_event(_event), do: :ok
99
+
end
100
+
101
+
# Example: Filter for specific DIDs
102
+
defmodule SpecificDIDConsumer do
103
+
@behaviour Drinkup.Jetstream.Consumer
104
+
105
+
@watched_dids [
106
+
"did:plc:abc123",
107
+
"did:plc:def456"
108
+
]
109
+
110
+
def handle_event(%Drinkup.Jetstream.Event.Commit{did: did} = event)
111
+
when did in @watched_dids do
112
+
IO.puts("Activity from watched DID: #{did}")
113
+
IO.inspect(event)
114
+
:ok
115
+
end
116
+
117
+
def handle_event(_event), do: :ok
118
+
end
-35
examples/multiple_consumers.ex
-35
examples/multiple_consumers.ex
···
1
-
defmodule PostDeleteConsumer do
2
-
use Drinkup.RecordConsumer, collections: ["app.bsky.feed.post"]
3
-
4
-
def handle_delete(record) do
5
-
IO.inspect(record, label: "update")
6
-
end
7
-
end
8
-
9
-
defmodule IdentityConsumer do
10
-
@behaviour Drinkup.Consumer
11
-
12
-
def handle_event(%Drinkup.Event.Identity{} = event) do
13
-
IO.inspect(event, label: "identity event")
14
-
end
15
-
16
-
def handle_event(_), do: :noop
17
-
end
18
-
19
-
defmodule ExampleSupervisor do
20
-
use Supervisor
21
-
22
-
def start_link(arg \\ []) do
23
-
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
24
-
end
25
-
26
-
@impl true
27
-
def init(_) do
28
-
children = [
29
-
{Drinkup, %{consumer: PostDeleteConsumer}},
30
-
{Drinkup, %{consumer: IdentityConsumer, name: :identities}}
31
-
]
32
-
33
-
Supervisor.init(children, strategy: :one_for_one)
34
-
end
35
-
end
-32
examples/record_consumer.ex
-32
examples/record_consumer.ex
···
1
-
defmodule ExampleRecordConsumer do
2
-
use Drinkup.RecordConsumer, collections: [~r/app\.bsky\.graph\..+/, "app.bsky.feed.post"]
3
-
4
-
def handle_create(record) do
5
-
IO.inspect(record, label: "create")
6
-
end
7
-
8
-
def handle_update(record) do
9
-
IO.inspect(record, label: "update")
10
-
end
11
-
12
-
def handle_delete(record) do
13
-
IO.inspect(record, label: "delete")
14
-
end
15
-
end
16
-
17
-
defmodule ExampleSupervisor do
18
-
use Supervisor
19
-
20
-
def start_link(arg \\ []) do
21
-
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
22
-
end
23
-
24
-
@impl true
25
-
def init(_) do
26
-
children = [
27
-
{Drinkup, %{consumer: ExampleRecordConsumer}}
28
-
]
29
-
30
-
Supervisor.init(children, strategy: :one_for_one)
31
-
end
32
-
end
+33
examples/tap/tap_consumer.ex
+33
examples/tap/tap_consumer.ex
···
1
+
defmodule TapConsumer do
2
+
@behaviour Drinkup.Tap.Consumer
3
+
4
+
def handle_event(%Drinkup.Tap.Event.Record{} = record) do
5
+
IO.inspect(record, label: "Tap record event")
6
+
end
7
+
8
+
def handle_event(%Drinkup.Tap.Event.Identity{} = identity) do
9
+
IO.inspect(identity, label: "Tap identity event")
10
+
end
11
+
end
12
+
13
+
defmodule TapExampleSupervisor do
14
+
use Supervisor
15
+
16
+
def start_link(arg \\ []) do
17
+
Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
18
+
end
19
+
20
+
@impl true
21
+
def init(_) do
22
+
children = [
23
+
{Drinkup.Tap,
24
+
%{
25
+
consumer: TapConsumer,
26
+
name: MyTap,
27
+
host: "http://localhost:2480"
28
+
}}
29
+
]
30
+
31
+
Supervisor.init(children, strategy: :one_for_one)
32
+
end
33
+
end
+3
-3
flake.lock
+3
-3
flake.lock
···
2
2
"nodes": {
3
3
"nixpkgs": {
4
4
"locked": {
5
-
"lastModified": 1748026106,
6
-
"narHash": "sha256-6m1Y3/4pVw1RWTsrkAK2VMYSzG4MMIj7sqUy7o8th1o=",
5
+
"lastModified": 1767640445,
6
+
"narHash": "sha256-UWYqmD7JFBEDBHWYcqE6s6c77pWdcU/i+bwD6XxMb8A=",
7
7
"owner": "nixos",
8
8
"repo": "nixpkgs",
9
-
"rev": "063f43f2dbdef86376cc29ad646c45c46e93234c",
9
+
"rev": "9f0c42f8bc7151b8e7e5840fb3bd454ad850d8c5",
10
10
"type": "github"
11
11
},
12
12
"original": {
-9
lib/consumer.ex
-9
lib/consumer.ex
-32
lib/drinkup.ex
-32
lib/drinkup.ex
···
1
-
defmodule Drinkup do
2
-
use Supervisor
3
-
alias Drinkup.Options
4
-
5
-
@dialyzer nowarn_function: {:init, 1}
6
-
@impl true
7
-
def init({%Options{name: name} = drinkup_options, supervisor_options}) do
8
-
children = [
9
-
{Task.Supervisor, name: {:via, Registry, {Drinkup.Registry, {name, Tasks}}}},
10
-
{Drinkup.Socket, drinkup_options}
11
-
]
12
-
13
-
Supervisor.start_link(
14
-
children,
15
-
supervisor_options ++ [name: {:via, Registry, {Drinkup.Registry, {name, Supervisor}}}]
16
-
)
17
-
end
18
-
19
-
@spec child_spec(Options.options()) :: Supervisor.child_spec()
20
-
def child_spec(%{} = options), do: child_spec({options, [strategy: :one_for_one]})
21
-
22
-
@spec child_spec({Options.options(), Keyword.t()}) :: Supervisor.child_spec()
23
-
def child_spec({drinkup_options, supervisor_options}) do
24
-
%{
25
-
id: Map.get(drinkup_options, :name, __MODULE__),
26
-
start: {__MODULE__, :init, [{Options.from(drinkup_options), supervisor_options}]},
27
-
type: :supervisor,
28
-
restart: :permanent,
29
-
shutdown: 500
30
-
}
31
-
end
32
-
end
-53
lib/event/account.ex
-53
lib/event/account.ex
···
1
-
defmodule Drinkup.Event.Account do
2
-
@moduledoc """
3
-
Struct for account events from the ATProto Firehose.
4
-
"""
5
-
6
-
use TypedStruct
7
-
8
-
@type status() ::
9
-
:takendown
10
-
| :suspended
11
-
| :deleted
12
-
| :deactivated
13
-
| :desynchronized
14
-
| :throttled
15
-
| String.t()
16
-
17
-
typedstruct enforce: true do
18
-
field :seq, integer()
19
-
field :did, String.t()
20
-
field :time, NaiveDateTime.t()
21
-
field :active, bool()
22
-
field :status, status(), enforce: false
23
-
end
24
-
25
-
@spec from(map()) :: t()
26
-
def from(%{"seq" => seq, "did" => did, "time" => time, "active" => active} = msg) do
27
-
status = recognise_status(Map.get(msg, "status"))
28
-
time = NaiveDateTime.from_iso8601!(time)
29
-
30
-
%__MODULE__{
31
-
seq: seq,
32
-
did: did,
33
-
time: time,
34
-
active: active,
35
-
status: status
36
-
}
37
-
end
38
-
39
-
@spec recognise_status(String.t()) :: status()
40
-
defp recognise_status(status)
41
-
when status in [
42
-
"takendown",
43
-
"suspended",
44
-
"deleted",
45
-
"deactivated",
46
-
"desynchronized",
47
-
"throttled"
48
-
],
49
-
do: String.to_atom(status)
50
-
51
-
defp recognise_status(status) when is_binary(status), do: status
52
-
defp recognise_status(nil), do: nil
53
-
end
-100
lib/event/commit.ex
-100
lib/event/commit.ex
···
1
-
defmodule Drinkup.Event.Commit do
2
-
@moduledoc """
3
-
Struct for commit events from the ATProto Firehose.
4
-
"""
5
-
6
-
# TODO: see atp specs
7
-
@type tid() :: String.t()
8
-
9
-
alias __MODULE__.RepoOp
10
-
use TypedStruct
11
-
12
-
typedstruct enforce: true do
13
-
field :seq, integer()
14
-
# DEPCREATED
15
-
field :rebase, bool()
16
-
# DEPRECATED
17
-
field :too_big, bool()
18
-
field :repo, String.t()
19
-
field :commit, binary()
20
-
field :rev, tid()
21
-
field :since, tid() | nil
22
-
field :blocks, CAR.Archive.t()
23
-
field :ops, list(RepoOp.t())
24
-
# DEPRECATED
25
-
field :blobs, list(binary())
26
-
field :prev_data, binary(), enforce: nil
27
-
field :time, NaiveDateTime.t()
28
-
end
29
-
30
-
@spec from(map()) :: t()
31
-
def from(
32
-
%{
33
-
"seq" => seq,
34
-
"rebase" => rebase,
35
-
"tooBig" => too_big,
36
-
"repo" => repo,
37
-
"commit" => commit,
38
-
"rev" => rev,
39
-
"since" => since,
40
-
"blocks" => %CBOR.Tag{value: blocks},
41
-
"ops" => ops,
42
-
"blobs" => blobs,
43
-
"time" => time
44
-
} = msg
45
-
) do
46
-
prev_data =
47
-
Map.get(msg, "prevData")
48
-
49
-
time = NaiveDateTime.from_iso8601!(time)
50
-
{:ok, blocks} = CAR.decode(blocks)
51
-
52
-
%__MODULE__{
53
-
seq: seq,
54
-
rebase: rebase,
55
-
too_big: too_big,
56
-
repo: repo,
57
-
commit: commit,
58
-
rev: rev,
59
-
since: since,
60
-
blocks: blocks,
61
-
ops: Enum.map(ops, &RepoOp.from(&1, blocks)),
62
-
blobs: blobs,
63
-
prev_data: prev_data,
64
-
time: time
65
-
}
66
-
end
67
-
68
-
defmodule RepoOp do
69
-
typedstruct enforce: true do
70
-
@type action() :: :create | :update | :delete | String.t()
71
-
72
-
field :action, action()
73
-
field :path, String.t()
74
-
field :cid, binary()
75
-
field :prev, binary(), enforce: false
76
-
field :record, map() | nil
77
-
end
78
-
79
-
@spec from(map(), CAR.Archive.t()) :: t()
80
-
def from(%{"action" => action, "path" => path, "cid" => cid} = op, %CAR.Archive{} = blocks) do
81
-
prev = Map.get(op, "prev")
82
-
record = CAR.Archive.get_block(blocks, cid)
83
-
84
-
%__MODULE__{
85
-
action: recognise_action(action),
86
-
path: path,
87
-
cid: cid,
88
-
prev: prev,
89
-
record: record
90
-
}
91
-
end
92
-
93
-
@spec recognise_action(String.t()) :: action()
94
-
defp recognise_action(action) when action in ["create", "update", "delete"],
95
-
do: String.to_atom(action)
96
-
97
-
defp recognise_action(action) when is_binary(action), do: action
98
-
defp recognise_action(nil), do: nil
99
-
end
100
-
end
-27
lib/event/identity.ex
-27
lib/event/identity.ex
···
1
-
defmodule Drinkup.Event.Identity do
2
-
@moduledoc """
3
-
Struct for identity events from the ATProto Firehose.
4
-
"""
5
-
6
-
use TypedStruct
7
-
8
-
typedstruct enforce: true do
9
-
field :seq, integer()
10
-
field :did, String.t()
11
-
field :time, NaiveDateTime.t()
12
-
field :handle, String.t() | nil
13
-
end
14
-
15
-
@spec from(map()) :: t()
16
-
def from(%{"seq" => seq, "did" => did, "time" => time} = msg) do
17
-
handle = Map.get(msg, "handle")
18
-
time = NaiveDateTime.from_iso8601!(time)
19
-
20
-
%__MODULE__{
21
-
seq: seq,
22
-
did: did,
23
-
time: time,
24
-
handle: handle
25
-
}
26
-
end
27
-
end
-22
lib/event/info.ex
-22
lib/event/info.ex
···
1
-
defmodule Drinkup.Event.Info do
2
-
@moduledoc """
3
-
Struct for info events from the ATProto Firehose.
4
-
"""
5
-
6
-
use TypedStruct
7
-
8
-
typedstruct enforce: true do
9
-
field :name, String.t()
10
-
field :message, String.t() | nil
11
-
end
12
-
13
-
@spec from(map()) :: t()
14
-
def from(%{"name" => name} = msg) do
15
-
message = Map.get(msg, "message")
16
-
17
-
%__MODULE__{
18
-
name: name,
19
-
message: message
20
-
}
21
-
end
22
-
end
-28
lib/event/sync.ex
-28
lib/event/sync.ex
···
1
-
defmodule Drinkup.Event.Sync do
2
-
@moduledoc """
3
-
Struct for sync events from the ATProto Firehose.
4
-
"""
5
-
6
-
use TypedStruct
7
-
8
-
typedstruct enforce: true do
9
-
field :seq, integer()
10
-
field :did, String.t()
11
-
field :blocks, binary()
12
-
field :rev, String.t()
13
-
field :time, NaiveDateTime.t()
14
-
end
15
-
16
-
@spec from(map()) :: t()
17
-
def from(%{"seq" => seq, "did" => did, "blocks" => blocks, "rev" => rev, "time" => time}) do
18
-
time = NaiveDateTime.from_iso8601!(time)
19
-
20
-
%__MODULE__{
21
-
seq: seq,
22
-
did: did,
23
-
blocks: blocks,
24
-
rev: rev,
25
-
time: time
26
-
}
27
-
end
28
-
end
-42
lib/event.ex
-42
lib/event.ex
···
1
-
defmodule Drinkup.Event do
2
-
require Logger
3
-
alias Drinkup.{Event, Options}
4
-
5
-
@type t() ::
6
-
Event.Commit.t()
7
-
| Event.Sync.t()
8
-
| Event.Identity.t()
9
-
| Event.Account.t()
10
-
| Event.Info.t()
11
-
12
-
@spec from(String.t(), map()) :: t() | nil
13
-
def from("#commit", payload), do: Event.Commit.from(payload)
14
-
def from("#sync", payload), do: Event.Sync.from(payload)
15
-
def from("#identity", payload), do: Event.Identity.from(payload)
16
-
def from("#account", payload), do: Event.Account.from(payload)
17
-
def from("#info", payload), do: Event.Info.from(payload)
18
-
def from(_type, _payload), do: nil
19
-
20
-
@spec valid_seq?(integer() | nil, any()) :: boolean()
21
-
def valid_seq?(nil, seq) when is_integer(seq), do: true
22
-
def valid_seq?(last_seq, nil) when is_integer(last_seq), do: true
23
-
def valid_seq?(last_seq, seq) when is_integer(last_seq) and is_integer(seq), do: seq > last_seq
24
-
def valid_seq?(_last_seq, _seq), do: false
25
-
26
-
@spec dispatch(t(), Options.t()) :: :ok
27
-
def dispatch(message, %Options{consumer: consumer, name: name}) do
28
-
supervisor_name = {:via, Registry, {Drinkup.Registry, {name, Tasks}}}
29
-
30
-
{:ok, _pid} =
31
-
Task.Supervisor.start_child(supervisor_name, fn ->
32
-
try do
33
-
consumer.handle_event(message)
34
-
rescue
35
-
e ->
36
-
Logger.error("Error in event handler: #{Exception.format(:error, e, __STACKTRACE__)}")
37
-
end
38
-
end)
39
-
40
-
:ok
41
-
end
42
-
end
+9
lib/firehose/consumer.ex
+9
lib/firehose/consumer.ex
+53
lib/firehose/event/account.ex
+53
lib/firehose/event/account.ex
···
1
+
defmodule Drinkup.Firehose.Event.Account do
2
+
@moduledoc """
3
+
Struct for account events from the ATProto Firehose.
4
+
"""
5
+
6
+
use TypedStruct
7
+
8
+
@type status() ::
9
+
:takendown
10
+
| :suspended
11
+
| :deleted
12
+
| :deactivated
13
+
| :desynchronized
14
+
| :throttled
15
+
| String.t()
16
+
17
+
typedstruct enforce: true do
18
+
field :seq, integer()
19
+
field :did, String.t()
20
+
field :time, NaiveDateTime.t()
21
+
field :active, bool()
22
+
field :status, status(), enforce: false
23
+
end
24
+
25
+
@spec from(map()) :: t()
26
+
def from(%{"seq" => seq, "did" => did, "time" => time, "active" => active} = msg) do
27
+
status = recognise_status(Map.get(msg, "status"))
28
+
time = NaiveDateTime.from_iso8601!(time)
29
+
30
+
%__MODULE__{
31
+
seq: seq,
32
+
did: did,
33
+
time: time,
34
+
active: active,
35
+
status: status
36
+
}
37
+
end
38
+
39
+
@spec recognise_status(String.t()) :: status()
40
+
defp recognise_status(status)
41
+
when status in [
42
+
"takendown",
43
+
"suspended",
44
+
"deleted",
45
+
"deactivated",
46
+
"desynchronized",
47
+
"throttled"
48
+
],
49
+
do: String.to_atom(status)
50
+
51
+
defp recognise_status(status) when is_binary(status), do: status
52
+
defp recognise_status(nil), do: nil
53
+
end
+100
lib/firehose/event/commit.ex
+100
lib/firehose/event/commit.ex
···
1
+
defmodule Drinkup.Firehose.Event.Commit do
2
+
@moduledoc """
3
+
Struct for commit events from the ATProto Firehose.
4
+
"""
5
+
6
+
# TODO: see atp specs
7
+
@type tid() :: String.t()
8
+
9
+
alias __MODULE__.RepoOp
10
+
use TypedStruct
11
+
12
+
typedstruct enforce: true do
13
+
field :seq, integer()
14
+
# DEPCREATED
15
+
field :rebase, bool()
16
+
# DEPRECATED
17
+
field :too_big, bool()
18
+
field :repo, String.t()
19
+
field :commit, binary()
20
+
field :rev, tid()
21
+
field :since, tid() | nil
22
+
field :blocks, CAR.Archive.t()
23
+
field :ops, list(RepoOp.t())
24
+
# DEPRECATED
25
+
field :blobs, list(binary())
26
+
field :prev_data, binary(), enforce: nil
27
+
field :time, NaiveDateTime.t()
28
+
end
29
+
30
+
@spec from(map()) :: t()
31
+
def from(
32
+
%{
33
+
"seq" => seq,
34
+
"rebase" => rebase,
35
+
"tooBig" => too_big,
36
+
"repo" => repo,
37
+
"commit" => commit,
38
+
"rev" => rev,
39
+
"since" => since,
40
+
"blocks" => %CBOR.Tag{value: blocks},
41
+
"ops" => ops,
42
+
"blobs" => blobs,
43
+
"time" => time
44
+
} = msg
45
+
) do
46
+
prev_data =
47
+
Map.get(msg, "prevData")
48
+
49
+
time = NaiveDateTime.from_iso8601!(time)
50
+
{:ok, blocks} = CAR.decode(blocks)
51
+
52
+
%__MODULE__{
53
+
seq: seq,
54
+
rebase: rebase,
55
+
too_big: too_big,
56
+
repo: repo,
57
+
commit: commit,
58
+
rev: rev,
59
+
since: since,
60
+
blocks: blocks,
61
+
ops: Enum.map(ops, &RepoOp.from(&1, blocks)),
62
+
blobs: blobs,
63
+
prev_data: prev_data,
64
+
time: time
65
+
}
66
+
end
67
+
68
+
defmodule RepoOp do
69
+
typedstruct enforce: true do
70
+
@type action() :: :create | :update | :delete | String.t()
71
+
72
+
field :action, action()
73
+
field :path, String.t()
74
+
field :cid, binary()
75
+
field :prev, binary(), enforce: false
76
+
field :record, map() | nil
77
+
end
78
+
79
+
@spec from(map(), CAR.Archive.t()) :: t()
80
+
def from(%{"action" => action, "path" => path, "cid" => cid} = op, %CAR.Archive{} = blocks) do
81
+
prev = Map.get(op, "prev")
82
+
record = CAR.Archive.get_block(blocks, cid)
83
+
84
+
%__MODULE__{
85
+
action: recognise_action(action),
86
+
path: path,
87
+
cid: cid,
88
+
prev: prev,
89
+
record: record
90
+
}
91
+
end
92
+
93
+
@spec recognise_action(String.t()) :: action()
94
+
defp recognise_action(action) when action in ["create", "update", "delete"],
95
+
do: String.to_atom(action)
96
+
97
+
defp recognise_action(action) when is_binary(action), do: action
98
+
defp recognise_action(nil), do: nil
99
+
end
100
+
end
+27
lib/firehose/event/identity.ex
+27
lib/firehose/event/identity.ex
···
1
+
defmodule Drinkup.Firehose.Event.Identity do
2
+
@moduledoc """
3
+
Struct for identity events from the ATProto Firehose.
4
+
"""
5
+
6
+
use TypedStruct
7
+
8
+
typedstruct enforce: true do
9
+
field :seq, integer()
10
+
field :did, String.t()
11
+
field :time, NaiveDateTime.t()
12
+
field :handle, String.t() | nil
13
+
end
14
+
15
+
@spec from(map()) :: t()
16
+
def from(%{"seq" => seq, "did" => did, "time" => time} = msg) do
17
+
handle = Map.get(msg, "handle")
18
+
time = NaiveDateTime.from_iso8601!(time)
19
+
20
+
%__MODULE__{
21
+
seq: seq,
22
+
did: did,
23
+
time: time,
24
+
handle: handle
25
+
}
26
+
end
27
+
end
+22
lib/firehose/event/info.ex
+22
lib/firehose/event/info.ex
···
1
+
defmodule Drinkup.Firehose.Event.Info do
2
+
@moduledoc """
3
+
Struct for info events from the ATProto Firehose.
4
+
"""
5
+
6
+
use TypedStruct
7
+
8
+
typedstruct enforce: true do
9
+
field :name, String.t()
10
+
field :message, String.t() | nil
11
+
end
12
+
13
+
@spec from(map()) :: t()
14
+
def from(%{"name" => name} = msg) do
15
+
message = Map.get(msg, "message")
16
+
17
+
%__MODULE__{
18
+
name: name,
19
+
message: message
20
+
}
21
+
end
22
+
end
+28
lib/firehose/event/sync.ex
+28
lib/firehose/event/sync.ex
···
1
+
defmodule Drinkup.Firehose.Event.Sync do
2
+
@moduledoc """
3
+
Struct for sync events from the ATProto Firehose.
4
+
"""
5
+
6
+
use TypedStruct
7
+
8
+
typedstruct enforce: true do
9
+
field :seq, integer()
10
+
field :did, String.t()
11
+
field :blocks, binary()
12
+
field :rev, String.t()
13
+
field :time, NaiveDateTime.t()
14
+
end
15
+
16
+
@spec from(map()) :: t()
17
+
def from(%{"seq" => seq, "did" => did, "blocks" => blocks, "rev" => rev, "time" => time}) do
18
+
time = NaiveDateTime.from_iso8601!(time)
19
+
20
+
%__MODULE__{
21
+
seq: seq,
22
+
did: did,
23
+
blocks: blocks,
24
+
rev: rev,
25
+
time: time
26
+
}
27
+
end
28
+
end
+42
lib/firehose/event.ex
+42
lib/firehose/event.ex
···
1
+
defmodule Drinkup.Firehose.Event do
2
+
require Logger
3
+
alias Drinkup.Firehose.{Event, Options}
4
+
5
+
@type t() ::
6
+
Event.Commit.t()
7
+
| Event.Sync.t()
8
+
| Event.Identity.t()
9
+
| Event.Account.t()
10
+
| Event.Info.t()
11
+
12
+
@spec from(String.t(), map()) :: t() | nil
13
+
def from("#commit", payload), do: Event.Commit.from(payload)
14
+
def from("#sync", payload), do: Event.Sync.from(payload)
15
+
def from("#identity", payload), do: Event.Identity.from(payload)
16
+
def from("#account", payload), do: Event.Account.from(payload)
17
+
def from("#info", payload), do: Event.Info.from(payload)
18
+
def from(_type, _payload), do: nil
19
+
20
+
@spec valid_seq?(integer() | nil, any()) :: boolean()
21
+
def valid_seq?(nil, seq) when is_integer(seq), do: true
22
+
def valid_seq?(last_seq, nil) when is_integer(last_seq), do: true
23
+
def valid_seq?(last_seq, seq) when is_integer(last_seq) and is_integer(seq), do: seq > last_seq
24
+
def valid_seq?(_last_seq, _seq), do: false
25
+
26
+
@spec dispatch(t(), Options.t()) :: :ok
27
+
def dispatch(message, %Options{consumer: consumer, name: name}) do
28
+
supervisor_name = {:via, Registry, {Drinkup.Registry, {name, Tasks}}}
29
+
30
+
{:ok, _pid} =
31
+
Task.Supervisor.start_child(supervisor_name, fn ->
32
+
try do
33
+
consumer.handle_event(message)
34
+
rescue
35
+
e ->
36
+
Logger.error("Error in event handler: #{Exception.format(:error, e, __STACKTRACE__)}")
37
+
end
38
+
end)
39
+
40
+
:ok
41
+
end
42
+
end
+79
lib/firehose/options.ex
+79
lib/firehose/options.ex
···
1
+
defmodule Drinkup.Firehose.Options do
2
+
@moduledoc """
3
+
Configuration options for ATProto Firehose relay subscriptions.
4
+
5
+
This module defines the configuration structure for connecting to and
6
+
consuming events from an ATProto Firehose relay. The Firehose streams
7
+
real-time repository events from the AT Protocol network.
8
+
9
+
## Options
10
+
11
+
- `:consumer` (required) - Module implementing `Drinkup.Firehose.Consumer` behaviour
12
+
- `:name` - Unique name for this Firehose instance in the supervision tree (default: `Drinkup.Firehose`)
13
+
- `:host` - Firehose relay URL (default: `"https://bsky.network"`)
14
+
- `:cursor` - Optional sequence number to resume streaming from
15
+
16
+
## Example
17
+
18
+
%{
19
+
consumer: MyFirehoseConsumer,
20
+
name: MyFirehose,
21
+
host: "https://bsky.network",
22
+
cursor: 12345
23
+
}
24
+
"""
25
+
26
+
use TypedStruct
27
+
28
+
@default_host "https://bsky.network"
29
+
30
+
@typedoc """
31
+
Map of configuration options accepted by `Drinkup.Firehose.child_spec/1`.
32
+
"""
33
+
@type options() :: %{
34
+
required(:consumer) => consumer(),
35
+
optional(:name) => name(),
36
+
optional(:host) => host(),
37
+
optional(:cursor) => cursor()
38
+
}
39
+
40
+
@typedoc """
41
+
Module implementing the `Drinkup.Firehose.Consumer` behaviour.
42
+
"""
43
+
@type consumer() :: module()
44
+
45
+
@typedoc """
46
+
Unique identifier for this Firehose instance in the supervision tree.
47
+
48
+
Used for Registry lookups and naming child processes.
49
+
"""
50
+
@type name() :: atom()
51
+
52
+
@typedoc """
53
+
HTTP/HTTPS URL of the ATProto Firehose relay.
54
+
55
+
Defaults to `"https://bsky.network"` which is the public Bluesky relay.
56
+
57
+
You can find a list of third-party relays at https://compare.hose.cam/.
58
+
"""
59
+
@type host() :: String.t()
60
+
61
+
@typedoc """
62
+
Optional sequence number to resume streaming from.
63
+
64
+
When provided, the Firehose will replay events starting from this sequence
65
+
number. Useful for resuming after a restart without missing events. The
66
+
cursor is automatically tracked and updated as events are received.
67
+
"""
68
+
@type cursor() :: pos_integer() | nil
69
+
70
+
typedstruct do
71
+
field :consumer, consumer(), enforce: true
72
+
field :name, name(), default: Drinkup.Firehose
73
+
field :host, host(), default: @default_host
74
+
field :cursor, cursor()
75
+
end
76
+
77
+
@spec from(options()) :: t()
78
+
def from(%{consumer: _} = options), do: struct(__MODULE__, options)
79
+
end
+85
lib/firehose/record_consumer.ex
+85
lib/firehose/record_consumer.ex
···
1
+
defmodule Drinkup.Firehose.RecordConsumer do
2
+
@moduledoc """
3
+
An opinionated consumer of the Firehose that eats consumers
4
+
"""
5
+
6
+
@callback handle_create(any()) :: any()
7
+
@callback handle_update(any()) :: any()
8
+
@callback handle_delete(any()) :: any()
9
+
10
+
defmacro __using__(opts) do
11
+
{collections, _opts} = Keyword.pop(opts, :collections, [])
12
+
13
+
quote location: :keep do
14
+
@behaviour Drinkup.Firehose.Consumer
15
+
@behaviour Drinkup.Firehose.RecordConsumer
16
+
17
+
def handle_event(%Drinkup.Firehose.Event.Commit{} = event) do
18
+
event.ops
19
+
|> Enum.filter(fn %{path: path} ->
20
+
path |> String.split("/") |> Enum.at(0) |> matches_collections?()
21
+
end)
22
+
|> Enum.map(&Drinkup.Firehose.RecordConsumer.Record.from(&1, event.repo))
23
+
|> Enum.each(&apply(__MODULE__, :"handle_#{&1.action}", [&1]))
24
+
end
25
+
26
+
def handle_event(_event), do: :noop
27
+
28
+
unquote(
29
+
if collections == [] do
30
+
quote do
31
+
def matches_collections?(_type), do: true
32
+
end
33
+
else
34
+
quote do
35
+
def matches_collections?(nil), do: false
36
+
37
+
def matches_collections?(type) when is_binary(type),
38
+
do:
39
+
Enum.any?(unquote(collections), fn
40
+
matcher when is_binary(matcher) -> type == matcher
41
+
matcher -> Regex.match?(matcher, type)
42
+
end)
43
+
end
44
+
end
45
+
)
46
+
47
+
@impl true
48
+
def handle_create(_record), do: nil
49
+
@impl true
50
+
def handle_update(_record), do: nil
51
+
@impl true
52
+
def handle_delete(_record), do: nil
53
+
54
+
defoverridable handle_create: 1, handle_update: 1, handle_delete: 1
55
+
end
56
+
end
57
+
58
+
defmodule Record do
59
+
alias Drinkup.Firehose.Event.Commit.RepoOp
60
+
use TypedStruct
61
+
62
+
typedstruct do
63
+
field :type, String.t()
64
+
field :rkey, String.t()
65
+
field :did, String.t()
66
+
field :action, :create | :update | :delete
67
+
field :cid, binary() | nil
68
+
field :record, map() | nil
69
+
end
70
+
71
+
@spec from(RepoOp.t(), String.t()) :: t()
72
+
def from(%RepoOp{action: action, path: path, cid: cid, record: record}, did) do
73
+
[type, rkey] = String.split(path, "/")
74
+
75
+
%__MODULE__{
76
+
type: type,
77
+
rkey: rkey,
78
+
did: did,
79
+
action: action,
80
+
cid: cid,
81
+
record: record
82
+
}
83
+
end
84
+
end
85
+
end
+90
lib/firehose/socket.ex
+90
lib/firehose/socket.ex
···
1
+
defmodule Drinkup.Firehose.Socket do
2
+
@moduledoc """
3
+
WebSocket connection handler for ATProto relay subscriptions.
4
+
5
+
Implements the Drinkup.Socket behaviour to manage connections to an ATProto
6
+
Firehose relay, handling CAR/CBOR-encoded frames and dispatching events to
7
+
the configured consumer.
8
+
"""
9
+
10
+
use Drinkup.Socket
11
+
12
+
require Logger
13
+
alias Drinkup.Firehose.{Event, Options}
14
+
15
+
@op_regular 1
16
+
@op_error -1
17
+
18
+
@impl true
19
+
def init(opts) do
20
+
options = Keyword.fetch!(opts, :options)
21
+
{:ok, %{seq: options.cursor, options: options, host: options.host}}
22
+
end
23
+
24
+
def start_link(%Options{} = options, statem_opts) do
25
+
# Build opts for Drinkup.Socket from Options struct
26
+
socket_opts = [
27
+
host: options.host,
28
+
cursor: options.cursor,
29
+
options: options
30
+
]
31
+
32
+
Drinkup.Socket.start_link(__MODULE__, socket_opts, statem_opts)
33
+
end
34
+
35
+
@impl true
36
+
def build_path(%{seq: seq}) do
37
+
cursor_param = if seq, do: %{cursor: seq}, else: %{}
38
+
"/xrpc/com.atproto.sync.subscribeRepos?" <> URI.encode_query(cursor_param)
39
+
end
40
+
41
+
@impl true
42
+
def handle_frame({:binary, frame}, {%{seq: seq, options: options} = data, _conn, _stream}) do
43
+
with {:ok, header, next} <- CAR.DagCbor.decode(frame),
44
+
{:ok, payload, _} <- CAR.DagCbor.decode(next),
45
+
{%{"op" => @op_regular, "t" => type}, _} <- {header, payload},
46
+
true <- Event.valid_seq?(seq, payload["seq"]) do
47
+
new_seq = payload["seq"] || seq
48
+
49
+
case Event.from(type, payload) do
50
+
nil ->
51
+
Logger.warning("Received unrecognised event from firehose: #{inspect({type, payload})}")
52
+
53
+
message ->
54
+
Event.dispatch(message, options)
55
+
end
56
+
57
+
{:ok, %{data | seq: new_seq}}
58
+
else
59
+
false ->
60
+
Logger.error("Got out of sequence or invalid `seq` from Firehose")
61
+
:noop
62
+
63
+
{%{"op" => @op_error, "t" => type}, payload} ->
64
+
Logger.error("Got error from Firehose: #{inspect({type, payload})}")
65
+
:noop
66
+
67
+
{:error, reason} ->
68
+
Logger.warning("Failed to decode frame from Firehose: #{inspect(reason)}")
69
+
:noop
70
+
end
71
+
end
72
+
73
+
@impl true
74
+
def handle_frame(:close, _data) do
75
+
Logger.info("Websocket closed, reason unknown")
76
+
nil
77
+
end
78
+
79
+
@impl true
80
+
def handle_frame({:close, errno, reason}, _data) do
81
+
Logger.info("Websocket closed, errno: #{errno}, reason: #{inspect(reason)}")
82
+
nil
83
+
end
84
+
85
+
@impl true
86
+
def handle_frame({:text, _text}, _data) do
87
+
Logger.warning("Received unexpected text frame from Firehose")
88
+
:noop
89
+
end
90
+
end
+32
lib/firehose.ex
+32
lib/firehose.ex
···
1
+
defmodule Drinkup.Firehose do
2
+
use Supervisor
3
+
alias Drinkup.Firehose.Options
4
+
5
+
@dialyzer nowarn_function: {:init, 1}
6
+
@impl true
7
+
def init({%Options{name: name} = drinkup_options, supervisor_options}) do
8
+
children = [
9
+
{Task.Supervisor, name: {:via, Registry, {Drinkup.Registry, {name, Tasks}}}},
10
+
{Drinkup.Firehose.Socket, drinkup_options}
11
+
]
12
+
13
+
Supervisor.start_link(
14
+
children,
15
+
supervisor_options ++ [name: {:via, Registry, {Drinkup.Registry, {name, Supervisor}}}]
16
+
)
17
+
end
18
+
19
+
@spec child_spec(Options.options()) :: Supervisor.child_spec()
20
+
def child_spec(%{} = options), do: child_spec({options, [strategy: :one_for_one]})
21
+
22
+
@spec child_spec({Options.options(), Keyword.t()}) :: Supervisor.child_spec()
23
+
def child_spec({drinkup_options, supervisor_options}) do
24
+
%{
25
+
id: Map.get(drinkup_options, :name, __MODULE__),
26
+
start: {__MODULE__, :init, [{Options.from(drinkup_options), supervisor_options}]},
27
+
type: :supervisor,
28
+
restart: :permanent,
29
+
shutdown: 500
30
+
}
31
+
end
32
+
end
+61
lib/jetstream/consumer.ex
+61
lib/jetstream/consumer.ex
···
1
+
defmodule Drinkup.Jetstream.Consumer do
2
+
@moduledoc """
3
+
Consumer behaviour for handling Jetstream events.
4
+
5
+
Implement this behaviour to process events from a Jetstream instance.
6
+
Events are dispatched asynchronously via `Task.Supervisor`.
7
+
8
+
Unlike Tap, Jetstream does not require event acknowledgments. Events are
9
+
processed in a fire-and-forget manner.
10
+
11
+
## Example
12
+
13
+
defmodule MyJetstreamConsumer do
14
+
@behaviour Drinkup.Jetstream.Consumer
15
+
16
+
def handle_event(%Drinkup.Jetstream.Event.Commit{operation: :create} = event) do
17
+
# Handle new record creation
18
+
IO.inspect(event, label: "New record")
19
+
:ok
20
+
end
21
+
22
+
def handle_event(%Drinkup.Jetstream.Event.Commit{operation: :delete} = event) do
23
+
# Handle record deletion
24
+
IO.inspect(event, label: "Deleted record")
25
+
:ok
26
+
end
27
+
28
+
def handle_event(%Drinkup.Jetstream.Event.Identity{} = event) do
29
+
# Handle identity changes
30
+
IO.inspect(event, label: "Identity update")
31
+
:ok
32
+
end
33
+
34
+
def handle_event(%Drinkup.Jetstream.Event.Account{active: false} = event) do
35
+
# Handle account deactivation
36
+
IO.inspect(event, label: "Account inactive")
37
+
:ok
38
+
end
39
+
40
+
def handle_event(_event), do: :ok
41
+
end
42
+
43
+
## Event Types
44
+
45
+
The consumer will receive one of three event types:
46
+
47
+
- `Drinkup.Jetstream.Event.Commit` - Repository commits (create, update, delete)
48
+
- `Drinkup.Jetstream.Event.Identity` - Identity updates (handle changes, etc.)
49
+
- `Drinkup.Jetstream.Event.Account` - Account status changes (active, taken down, etc.)
50
+
51
+
## Error Handling
52
+
53
+
If your `handle_event/1` implementation raises an exception, it will be logged
54
+
but will not affect the stream. The error is caught and logged by the event
55
+
dispatcher.
56
+
"""
57
+
58
+
alias Drinkup.Jetstream.Event
59
+
60
+
@callback handle_event(Event.t()) :: any()
61
+
end
+106
lib/jetstream/event/account.ex
+106
lib/jetstream/event/account.ex
···
1
+
defmodule Drinkup.Jetstream.Event.Account do
2
+
@moduledoc """
3
+
Struct for account events from Jetstream.
4
+
5
+
Represents a change to an account's status on a host (e.g., PDS or Relay).
6
+
The semantics of this event are that the status is at the host which emitted
7
+
the event, not necessarily that at the currently active PDS.
8
+
9
+
For example, a Relay takedown would emit a takedown with `active: false`,
10
+
even if the PDS is still active.
11
+
"""
12
+
13
+
use TypedStruct
14
+
15
+
typedstruct enforce: true do
16
+
@typedoc """
17
+
The status of an inactive account.
18
+
19
+
Known values from the ATProto lexicon:
20
+
- `:takendown` - Account has been taken down
21
+
- `:suspended` - Account is suspended
22
+
- `:deleted` - Account has been deleted
23
+
- `:deactivated` - Account has been deactivated by the user
24
+
- `:desynchronized` - Account is out of sync
25
+
- `:throttled` - Account is throttled
26
+
27
+
The status can also be any other string value for future compatibility.
28
+
"""
29
+
@type status() ::
30
+
:takendown
31
+
| :suspended
32
+
| :deleted
33
+
| :deactivated
34
+
| :desynchronized
35
+
| :throttled
36
+
| String.t()
37
+
38
+
field :did, String.t()
39
+
field :time_us, integer()
40
+
field :kind, :account, default: :account
41
+
field :active, boolean()
42
+
field :seq, integer()
43
+
field :time, NaiveDateTime.t()
44
+
field :status, status() | nil
45
+
end
46
+
47
+
@doc """
48
+
Parses a Jetstream account payload into an Account struct.
49
+
50
+
## Example Payload (Active)
51
+
52
+
%{
53
+
"active" => true,
54
+
"did" => "did:plc:ufbl4k27gp6kzas5glhz7fim",
55
+
"seq" => 1409753013,
56
+
"time" => "2024-09-05T06:11:04.870Z"
57
+
}
58
+
59
+
## Example Payload (Inactive)
60
+
61
+
%{
62
+
"active" => false,
63
+
"did" => "did:plc:abc123",
64
+
"seq" => 1409753014,
65
+
"time" => "2024-09-05T06:12:00.000Z",
66
+
"status" => "takendown"
67
+
}
68
+
"""
69
+
@spec from(String.t(), integer(), map()) :: t()
70
+
def from(
71
+
did,
72
+
time_us,
73
+
%{
74
+
"active" => active,
75
+
"seq" => seq,
76
+
"time" => time
77
+
} = account
78
+
) do
79
+
%__MODULE__{
80
+
did: did,
81
+
time_us: time_us,
82
+
active: active,
83
+
seq: seq,
84
+
time: parse_datetime(time),
85
+
status: parse_status(Map.get(account, "status"))
86
+
}
87
+
end
88
+
89
+
@spec parse_datetime(String.t()) :: NaiveDateTime.t()
90
+
defp parse_datetime(time_str) do
91
+
case NaiveDateTime.from_iso8601(time_str) do
92
+
{:ok, datetime} -> datetime
93
+
{:error, _} -> raise "Invalid datetime format: #{time_str}"
94
+
end
95
+
end
96
+
97
+
@spec parse_status(String.t() | nil) :: status() | nil
98
+
defp parse_status(nil), do: nil
99
+
defp parse_status("takendown"), do: :takendown
100
+
defp parse_status("suspended"), do: :suspended
101
+
defp parse_status("deleted"), do: :deleted
102
+
defp parse_status("deactivated"), do: :deactivated
103
+
defp parse_status("desynchronized"), do: :desynchronized
104
+
defp parse_status("throttled"), do: :throttled
105
+
defp parse_status(status) when is_binary(status), do: status
106
+
end
+78
lib/jetstream/event/commit.ex
+78
lib/jetstream/event/commit.ex
···
1
+
defmodule Drinkup.Jetstream.Event.Commit do
2
+
@moduledoc """
3
+
Struct for commit events from Jetstream.
4
+
5
+
Represents a repository commit containing either a create, update, or delete
6
+
operation on a record. Unlike the Firehose commit events, Jetstream provides
7
+
simplified JSON structures without CAR/CBOR encoding.
8
+
"""
9
+
10
+
use TypedStruct
11
+
12
+
typedstruct enforce: true do
13
+
@typedoc """
14
+
The operation type for this commit.
15
+
16
+
- `:create` - A new record was created
17
+
- `:update` - An existing record was updated
18
+
- `:delete` - An existing record was deleted
19
+
"""
20
+
@type operation() :: :create | :update | :delete
21
+
22
+
field :did, String.t()
23
+
field :time_us, integer()
24
+
field :kind, :commit, default: :commit
25
+
field :operation, operation()
26
+
field :collection, String.t()
27
+
field :rkey, String.t()
28
+
field :rev, String.t()
29
+
field :record, map() | nil
30
+
field :cid, String.t() | nil
31
+
end
32
+
33
+
@doc """
34
+
Parses a Jetstream commit payload into a Commit struct.
35
+
36
+
## Example Payload
37
+
38
+
%{
39
+
"rev" => "3l3qo2vutsw2b",
40
+
"operation" => "create",
41
+
"collection" => "app.bsky.feed.like",
42
+
"rkey" => "3l3qo2vuowo2b",
43
+
"record" => %{
44
+
"$type" => "app.bsky.feed.like",
45
+
"createdAt" => "2024-09-09T19:46:02.102Z",
46
+
"subject" => %{...}
47
+
},
48
+
"cid" => "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi"
49
+
}
50
+
"""
51
+
@spec from(String.t(), integer(), map()) :: t()
52
+
def from(
53
+
did,
54
+
time_us,
55
+
%{
56
+
"rev" => rev,
57
+
"operation" => operation,
58
+
"collection" => collection,
59
+
"rkey" => rkey
60
+
} = commit
61
+
) do
62
+
%__MODULE__{
63
+
did: did,
64
+
time_us: time_us,
65
+
operation: parse_operation(operation),
66
+
collection: collection,
67
+
rkey: rkey,
68
+
rev: rev,
69
+
record: Map.get(commit, "record"),
70
+
cid: Map.get(commit, "cid")
71
+
}
72
+
end
73
+
74
+
@spec parse_operation(String.t()) :: operation()
75
+
defp parse_operation("create"), do: :create
76
+
defp parse_operation("update"), do: :update
77
+
defp parse_operation("delete"), do: :delete
78
+
end
+58
lib/jetstream/event/identity.ex
+58
lib/jetstream/event/identity.ex
···
1
+
defmodule Drinkup.Jetstream.Event.Identity do
2
+
@moduledoc """
3
+
Struct for identity events from Jetstream.
4
+
5
+
Represents a change to an account's identity, such as an updated handle,
6
+
signing key, or PDS hosting endpoint. This serves as a signal to downstream
7
+
services to refresh their identity cache.
8
+
"""
9
+
10
+
use TypedStruct
11
+
12
+
typedstruct enforce: true do
13
+
field :did, String.t()
14
+
field :time_us, integer()
15
+
field :kind, :identity, default: :identity
16
+
field :handle, String.t() | nil
17
+
field :seq, integer()
18
+
field :time, NaiveDateTime.t()
19
+
end
20
+
21
+
@doc """
22
+
Parses a Jetstream identity payload into an Identity struct.
23
+
24
+
## Example Payload
25
+
26
+
%{
27
+
"did" => "did:plc:ufbl4k27gp6kzas5glhz7fim",
28
+
"handle" => "yohenrique.bsky.social",
29
+
"seq" => 1409752997,
30
+
"time" => "2024-09-05T06:11:04.870Z"
31
+
}
32
+
"""
33
+
@spec from(String.t(), integer(), map()) :: t()
34
+
def from(
35
+
did,
36
+
time_us,
37
+
%{
38
+
"seq" => seq,
39
+
"time" => time
40
+
} = identity
41
+
) do
42
+
%__MODULE__{
43
+
did: did,
44
+
time_us: time_us,
45
+
handle: Map.get(identity, "handle"),
46
+
seq: seq,
47
+
time: parse_datetime(time)
48
+
}
49
+
end
50
+
51
+
@spec parse_datetime(String.t()) :: NaiveDateTime.t()
52
+
defp parse_datetime(time_str) do
53
+
case NaiveDateTime.from_iso8601(time_str) do
54
+
{:ok, datetime} -> datetime
55
+
{:error, _} -> raise "Invalid datetime format: #{time_str}"
56
+
end
57
+
end
58
+
end
+100
lib/jetstream/event.ex
+100
lib/jetstream/event.ex
···
1
+
defmodule Drinkup.Jetstream.Event do
2
+
@moduledoc """
3
+
Event handling and dispatch for Jetstream events.
4
+
5
+
Parses incoming JSON events from Jetstream and dispatches them to the
6
+
configured consumer via Task.Supervisor.
7
+
"""
8
+
9
+
require Logger
10
+
alias Drinkup.Jetstream.{Event, Options}
11
+
12
+
@type t() :: Event.Commit.t() | Event.Identity.t() | Event.Account.t()
13
+
14
+
@doc """
15
+
Parse a JSON map into an event struct.
16
+
17
+
Jetstream events have a top-level structure with a "kind" field that
18
+
determines the event type, and a nested object with the event data.
19
+
20
+
## Example Event Structure
21
+
22
+
%{
23
+
"did" => "did:plc:...",
24
+
"time_us" => 1726880765818347,
25
+
"kind" => "commit",
26
+
"commit" => %{...}
27
+
}
28
+
29
+
Returns the appropriate event struct based on the "kind" field, or `nil`
30
+
if the event type is not recognized.
31
+
"""
32
+
@spec from(map()) :: t() | nil
33
+
def from(%{"did" => did, "time_us" => time_us, "kind" => kind} = payload) do
34
+
case kind do
35
+
"commit" ->
36
+
case Map.get(payload, "commit") do
37
+
nil ->
38
+
Logger.warning("Commit event missing 'commit' field: #{inspect(payload)}")
39
+
nil
40
+
41
+
commit ->
42
+
Event.Commit.from(did, time_us, commit)
43
+
end
44
+
45
+
"identity" ->
46
+
case Map.get(payload, "identity") do
47
+
nil ->
48
+
Logger.warning("Identity event missing 'identity' field: #{inspect(payload)}")
49
+
nil
50
+
51
+
identity ->
52
+
Event.Identity.from(did, time_us, identity)
53
+
end
54
+
55
+
"account" ->
56
+
case Map.get(payload, "account") do
57
+
nil ->
58
+
Logger.warning("Account event missing 'account' field: #{inspect(payload)}")
59
+
nil
60
+
61
+
account ->
62
+
Event.Account.from(did, time_us, account)
63
+
end
64
+
65
+
_ ->
66
+
Logger.warning("Received unrecognized event kind from Jetstream: #{inspect(kind)}")
67
+
nil
68
+
end
69
+
end
70
+
71
+
def from(payload) do
72
+
Logger.warning("Received invalid event structure from Jetstream: #{inspect(payload)}")
73
+
nil
74
+
end
75
+
76
+
@doc """
77
+
Dispatch an event to the consumer via Task.Supervisor.
78
+
79
+
Spawns a task that processes the event via the consumer's `handle_event/1`
80
+
callback. Unlike Tap, Jetstream does not require acknowledgments.
81
+
"""
82
+
@spec dispatch(t(), Options.t()) :: :ok
83
+
def dispatch(event, %Options{consumer: consumer, name: name}) do
84
+
supervisor_name = {:via, Registry, {Drinkup.Registry, {name, JetstreamTasks}}}
85
+
86
+
{:ok, _pid} =
87
+
Task.Supervisor.start_child(supervisor_name, fn ->
88
+
try do
89
+
consumer.handle_event(event)
90
+
rescue
91
+
e ->
92
+
Logger.error(
93
+
"Error in Jetstream event handler: #{Exception.format(:error, e, __STACKTRACE__)}"
94
+
)
95
+
end
96
+
end)
97
+
98
+
:ok
99
+
end
100
+
end
+151
lib/jetstream/options.ex
+151
lib/jetstream/options.ex
···
1
+
defmodule Drinkup.Jetstream.Options do
2
+
@moduledoc """
3
+
Configuration options for Jetstream event stream connection.
4
+
5
+
Jetstream is a simplified JSON event stream that converts the CBOR-encoded
6
+
ATProto Firehose into lightweight, friendly JSON. It provides zstd compression
7
+
and filtering capabilities for collections and DIDs.
8
+
9
+
## Options
10
+
11
+
- `:consumer` (required) - Module implementing `Drinkup.Jetstream.Consumer` behaviour
12
+
- `:name` - Unique name for this Jetstream instance in the supervision tree (default: `Drinkup.Jetstream`)
13
+
- `:host` - Jetstream service URL (default: `"wss://jetstream2.us-east.bsky.network"`)
14
+
- `:wanted_collections` - List of collection NSIDs or prefixes to filter (default: `[]` = all collections)
15
+
- `:wanted_dids` - List of DIDs to filter (default: `[]` = all repos)
16
+
- `:cursor` - Unix microseconds timestamp to resume from (default: `nil` = live-tail)
17
+
- `:require_hello` - Pause replay until first options update is sent (default: `false`)
18
+
- `:max_message_size_bytes` - Maximum message size to receive (default: `nil` = no limit)
19
+
20
+
## Example
21
+
22
+
%{
23
+
consumer: MyJetstreamConsumer,
24
+
name: MyJetstream,
25
+
host: "wss://jetstream2.us-east.bsky.network",
26
+
wanted_collections: ["app.bsky.feed.post", "app.bsky.feed.like"],
27
+
wanted_dids: ["did:plc:abc123"],
28
+
cursor: 1725519626134432
29
+
}
30
+
31
+
## Collection Filters
32
+
33
+
The `wanted_collections` option supports:
34
+
- Full NSIDs: `"app.bsky.feed.post"`
35
+
- NSID prefixes: `"app.bsky.graph.*"`, `"app.bsky.*"`
36
+
37
+
You can specify up to 100 collection filters.
38
+
39
+
## DID Filters
40
+
41
+
The `wanted_dids` option accepts a list of DID strings.
42
+
You can specify up to 10,000 DIDs.
43
+
44
+
## Compression
45
+
46
+
Jetstream always uses zstd compression with a custom dictionary.
47
+
This is handled automatically by the socket implementation.
48
+
"""
49
+
50
+
use TypedStruct
51
+
52
+
@default_host "wss://jetstream2.us-east.bsky.network"
53
+
54
+
@typedoc """
55
+
Map of configuration options accepted by `Drinkup.Jetstream.child_spec/1`.
56
+
"""
57
+
@type options() :: %{
58
+
required(:consumer) => consumer(),
59
+
optional(:name) => name(),
60
+
optional(:host) => host(),
61
+
optional(:wanted_collections) => wanted_collections(),
62
+
optional(:wanted_dids) => wanted_dids(),
63
+
optional(:cursor) => cursor(),
64
+
optional(:require_hello) => require_hello(),
65
+
optional(:max_message_size_bytes) => max_message_size_bytes()
66
+
}
67
+
68
+
@typedoc """
69
+
Module implementing the `Drinkup.Jetstream.Consumer` behaviour.
70
+
"""
71
+
@type consumer() :: module()
72
+
73
+
@typedoc """
74
+
Unique identifier for this Jetstream instance in the supervision tree.
75
+
76
+
Used for Registry lookups and naming child processes.
77
+
"""
78
+
@type name() :: atom()
79
+
80
+
@typedoc """
81
+
WebSocket URL of the Jetstream service.
82
+
83
+
Defaults to `"wss://jetstream2.us-east.bsky.network"` which is a public Bluesky instance.
84
+
"""
85
+
@type host() :: String.t()
86
+
87
+
@typedoc """
88
+
List of collection NSIDs or NSID prefixes to filter.
89
+
90
+
Examples:
91
+
- `["app.bsky.feed.post"]` - Only posts
92
+
- `["app.bsky.graph.*"]` - All graph collections
93
+
- `["app.bsky.*"]` - All Bluesky app collections
94
+
95
+
You can specify up to 100 collection filters.
96
+
Defaults to `[]` (all collections).
97
+
"""
98
+
@type wanted_collections() :: [String.t()]
99
+
100
+
@typedoc """
101
+
List of DIDs to filter events by.
102
+
103
+
You can specify up to 10,000 DIDs.
104
+
Defaults to `[]` (all repos).
105
+
"""
106
+
@type wanted_dids() :: [String.t()]
107
+
108
+
@typedoc """
109
+
Unix microseconds timestamp to resume streaming from.
110
+
111
+
When provided, Jetstream will replay events starting from this timestamp.
112
+
Useful for resuming after a restart without missing events. The cursor is
113
+
automatically tracked and updated as events are received.
114
+
115
+
Defaults to `nil` (live-tail from current time).
116
+
"""
117
+
@type cursor() :: pos_integer() | nil
118
+
119
+
@typedoc """
120
+
Whether to pause replay/live-tail until the first options update is sent.
121
+
122
+
When `true`, the connection will wait for a `Drinkup.Jetstream.update_options/2`
123
+
call before starting to receive events.
124
+
125
+
Defaults to `false`.
126
+
"""
127
+
@type require_hello() :: boolean()
128
+
129
+
@typedoc """
130
+
Maximum message size in bytes that the client would like to receive.
131
+
132
+
Zero or `nil` means no limit. Negative values are treated as zero.
133
+
Defaults to `nil` (no maximum size).
134
+
"""
135
+
@type max_message_size_bytes() :: integer() | nil
136
+
137
+
typedstruct do
138
+
field :consumer, consumer(), enforce: true
139
+
field :name, name(), default: Drinkup.Jetstream
140
+
field :host, host(), default: @default_host
141
+
# TODO: Add NSID prefix validation once available in atex
142
+
field :wanted_collections, wanted_collections(), default: []
143
+
field :wanted_dids, wanted_dids(), default: []
144
+
field :cursor, cursor()
145
+
field :require_hello, require_hello(), default: false
146
+
field :max_message_size_bytes, max_message_size_bytes()
147
+
end
148
+
149
+
@spec from(options()) :: t()
150
+
def from(%{consumer: _} = options), do: struct(__MODULE__, options)
151
+
end
+201
lib/jetstream/socket.ex
+201
lib/jetstream/socket.ex
···
1
+
defmodule Drinkup.Jetstream.Socket do
2
+
@moduledoc """
3
+
WebSocket connection handler for Jetstream event streams.
4
+
5
+
Implements the Drinkup.Socket behaviour to manage connections to a Jetstream
6
+
service, handling zstd-compressed JSON events and dispatching them to the
7
+
configured consumer.
8
+
"""
9
+
10
+
use Drinkup.Socket
11
+
12
+
require Logger
13
+
alias Drinkup.Jetstream.{Event, Options}
14
+
15
+
@dict_path "priv/jetstream/zstd_dictionary"
16
+
@external_resource @dict_path
17
+
@zstd_dict File.read!(@dict_path)
18
+
19
+
@impl true
20
+
def init(opts) do
21
+
options = Keyword.fetch!(opts, :options)
22
+
23
+
{:ok, %{options: options, host: options.host, cursor: options.cursor}}
24
+
end
25
+
26
+
def start_link(%Options{} = options, statem_opts) do
27
+
socket_opts = [
28
+
host: options.host,
29
+
options: options
30
+
]
31
+
32
+
statem_opts =
33
+
Keyword.put(
34
+
statem_opts,
35
+
:name,
36
+
{:via, Registry, {Drinkup.Registry, {options.name, JetstreamSocket}}}
37
+
)
38
+
39
+
Drinkup.Socket.start_link(__MODULE__, socket_opts, statem_opts)
40
+
end
41
+
42
+
@impl true
43
+
def build_path(%{options: options}) do
44
+
query_params = [compress: "true"]
45
+
46
+
query_params =
47
+
query_params
48
+
|> put_collections(options.wanted_collections)
49
+
|> put_dids(options.wanted_dids)
50
+
|> put_cursor(options.cursor)
51
+
|> put_max_size(options.max_message_size_bytes)
52
+
|> put_require_hello(options.require_hello)
53
+
54
+
"/subscribe?" <> URI.encode_query(query_params)
55
+
end
56
+
57
+
@impl true
58
+
def handle_frame(
59
+
{:binary, compressed_data},
60
+
{%{options: options} = data, _conn, _stream}
61
+
) do
62
+
case decompress_and_parse(compressed_data) do
63
+
{:ok, payload} ->
64
+
case Event.from(payload) do
65
+
nil ->
66
+
# Event.from already logs warnings for unrecognized events
67
+
:noop
68
+
69
+
event ->
70
+
Event.dispatch(event, options)
71
+
# Update cursor with the event's time_us
72
+
new_cursor = Map.get(payload, "time_us")
73
+
{:ok, %{data | cursor: new_cursor}}
74
+
end
75
+
76
+
# TODO: sometimes getting ZSTD_CONTENTSIZE_UNKNOWN
77
+
{:error, reason} ->
78
+
Logger.error(
79
+
"[Drinkup.Jetstream.Socket] Failed to decompress/parse frame: #{inspect(reason)}"
80
+
)
81
+
82
+
:noop
83
+
end
84
+
end
85
+
86
+
@impl true
87
+
def handle_frame({:text, json}, {%{options: options} = data, _conn, _stream}) do
88
+
# Text frames shouldn't happen since we force compression, but handle them anyway
89
+
case Jason.decode(json) do
90
+
{:ok, payload} ->
91
+
case Event.from(payload) do
92
+
nil ->
93
+
:noop
94
+
95
+
event ->
96
+
Event.dispatch(event, options)
97
+
new_cursor = Map.get(payload, "time_us")
98
+
{:ok, %{data | cursor: new_cursor}}
99
+
end
100
+
101
+
{:error, reason} ->
102
+
Logger.error("[Drinkup.Jetstream.Socket] Failed to decode JSON: #{inspect(reason)}")
103
+
:noop
104
+
end
105
+
end
106
+
107
+
@impl true
108
+
def handle_frame(:close, _data) do
109
+
Logger.info("[Drinkup.Jetstream.Socket] WebSocket closed, reason unknown")
110
+
nil
111
+
end
112
+
113
+
@impl true
114
+
def handle_frame({:close, errno, reason}, _data) do
115
+
Logger.info(
116
+
"[Drinkup.Jetstream.Socket] WebSocket closed, errno: #{errno}, reason: #{inspect(reason)}"
117
+
)
118
+
119
+
nil
120
+
end
121
+
122
+
@impl true
123
+
def handle_connected({user_data, conn, stream}) do
124
+
# Register connection for options updates
125
+
Registry.register(
126
+
Drinkup.Registry,
127
+
{user_data.options.name, JetstreamConnection},
128
+
{conn, stream}
129
+
)
130
+
131
+
{:ok, user_data}
132
+
end
133
+
134
+
@impl true
135
+
def handle_disconnected(_reason, {user_data, _conn, _stream}) do
136
+
# Unregister connection when disconnected
137
+
Registry.unregister(Drinkup.Registry, {user_data.options.name, JetstreamConnection})
138
+
{:ok, user_data}
139
+
end
140
+
141
+
# Can't use `create_ddict` as the value of `@zstd_dict` because it returns a reference :(
142
+
@spec get_dictionary() :: reference()
143
+
defp get_dictionary() do
144
+
case :ezstd.create_ddict(@zstd_dict) do
145
+
{:error, reason} ->
146
+
raise ArgumentError,
147
+
"somehow failed to created Jetstream's ZSTD dictionary: #{inspect(reason)}"
148
+
149
+
dict ->
150
+
dict
151
+
end
152
+
end
153
+
154
+
@spec decompress_and_parse(binary()) :: {:ok, map()} | {:error, term()}
155
+
defp decompress_and_parse(compressed_data) do
156
+
with ctx when is_reference(ctx) <-
157
+
:ezstd.create_decompression_context(byte_size(compressed_data)),
158
+
:ok <- :ezstd.select_ddict(ctx, get_dictionary()),
159
+
iolist when is_list(iolist) <- :ezstd.decompress_streaming(ctx, compressed_data),
160
+
decompressed <- IO.iodata_to_binary(iolist),
161
+
{:ok, payload} <- JSON.decode(decompressed) do
162
+
{:ok, payload}
163
+
else
164
+
{:error, reason} -> {:error, reason}
165
+
end
166
+
end
167
+
168
+
@spec put_collections(keyword(), [String.t()]) :: keyword()
169
+
defp put_collections(params, []), do: params
170
+
171
+
defp put_collections(params, collections) when is_list(collections) do
172
+
Enum.reduce(collections, params, fn collection, acc ->
173
+
[{:wantedCollections, collection} | acc]
174
+
end)
175
+
end
176
+
177
+
@spec put_dids(keyword(), [String.t()]) :: keyword()
178
+
defp put_dids(params, []), do: params
179
+
180
+
defp put_dids(params, dids) when is_list(dids) do
181
+
Enum.reduce(dids, params, fn did, acc ->
182
+
[{:wantedDids, did} | acc]
183
+
end)
184
+
end
185
+
186
+
@spec put_cursor(keyword(), integer() | nil) :: keyword()
187
+
defp put_cursor(params, nil), do: params
188
+
189
+
defp put_cursor(params, cursor) when is_integer(cursor), do: [{:cursor, cursor} | params]
190
+
191
+
@spec put_max_size(keyword(), integer() | nil) :: keyword()
192
+
defp put_max_size(params, nil), do: params
193
+
194
+
defp put_max_size(params, max_size) when is_integer(max_size),
195
+
do: [{:maxMessageSizeBytes, max_size} | params]
196
+
197
+
@spec put_require_hello(keyword(), boolean()) :: keyword()
198
+
defp put_require_hello(params, false), do: params
199
+
200
+
defp put_require_hello(params, true), do: [{:requireHello, "true"} | params]
201
+
end
+207
lib/jetstream.ex
+207
lib/jetstream.ex
···
1
+
defmodule Drinkup.Jetstream do
2
+
@moduledoc """
3
+
Supervisor for Jetstream event stream connections.
4
+
5
+
Jetstream is a simplified JSON event stream that converts the CBOR-encoded
6
+
ATProto Firehose into lightweight, friendly JSON events. It provides zstd
7
+
compression and filtering capabilities for collections and DIDs.
8
+
9
+
## Usage
10
+
11
+
Add Jetstream to your supervision tree:
12
+
13
+
children = [
14
+
{Drinkup.Jetstream, %{
15
+
consumer: MyJetstreamConsumer,
16
+
name: MyJetstream,
17
+
wanted_collections: ["app.bsky.feed.post", "app.bsky.feed.like"]
18
+
}}
19
+
]
20
+
21
+
## Configuration
22
+
23
+
See `Drinkup.Jetstream.Options` for all available configuration options.
24
+
25
+
## Dynamic Filter Updates
26
+
27
+
You can update filters after the connection is established:
28
+
29
+
Drinkup.Jetstream.update_options(MyJetstream, %{
30
+
wanted_collections: ["app.bsky.graph.follow"],
31
+
wanted_dids: ["did:plc:abc123"]
32
+
})
33
+
34
+
## Public Instances
35
+
36
+
By default Drinkup connects to `jetstream2.us-east.bsky.network`.
37
+
38
+
Bluesky operates a few different Jetstream instances:
39
+
- `jetstream1.us-east.bsky.network`
40
+
- `jetstream2.us-east.bsky.network`
41
+
- `jetstream1.us-west.bsky.network`
42
+
- `jetstream2.us-west.bsky.network`
43
+
44
+
There also some third-party instances not run by Bluesky PBC:
45
+
- `jetstream.fire.hose.cam`
46
+
- `jetstream2.fr.hose.cam`
47
+
- `jetstream1.us-east.fire.hose.cam`
48
+
"""
49
+
50
+
use Supervisor
51
+
require Logger
52
+
alias Drinkup.Jetstream.Options
53
+
54
+
@dialyzer nowarn_function: {:init, 1}
55
+
56
+
@impl true
57
+
def init({%Options{name: name} = drinkup_options, supervisor_options}) do
58
+
children = [
59
+
{Task.Supervisor, name: {:via, Registry, {Drinkup.Registry, {name, JetstreamTasks}}}},
60
+
{Drinkup.Jetstream.Socket, drinkup_options}
61
+
]
62
+
63
+
Supervisor.start_link(
64
+
children,
65
+
supervisor_options ++
66
+
[name: {:via, Registry, {Drinkup.Registry, {name, JetstreamSupervisor}}}]
67
+
)
68
+
end
69
+
70
+
@spec child_spec(Options.options()) :: Supervisor.child_spec()
71
+
def child_spec(%{} = options), do: child_spec({options, [strategy: :one_for_one]})
72
+
73
+
@spec child_spec({Options.options(), Keyword.t()}) :: Supervisor.child_spec()
74
+
def child_spec({drinkup_options, supervisor_options}) do
75
+
%{
76
+
id: Map.get(drinkup_options, :name, __MODULE__),
77
+
start: {__MODULE__, :init, [{Options.from(drinkup_options), supervisor_options}]},
78
+
type: :supervisor,
79
+
restart: :permanent,
80
+
shutdown: 500
81
+
}
82
+
end
83
+
84
+
# Options Update API
85
+
86
+
@typedoc """
87
+
Options that can be updated dynamically via `update_options/2`.
88
+
89
+
- `:wanted_collections` - List of collection NSIDs or prefixes (max 100)
90
+
- `:wanted_dids` - List of DIDs to filter (max 10,000)
91
+
- `:max_message_size_bytes` - Maximum message size to receive
92
+
93
+
Empty arrays will disable the corresponding filter (i.e., receive all).
94
+
"""
95
+
@type update_opts :: %{
96
+
optional(:wanted_collections) => [String.t()],
97
+
optional(:wanted_dids) => [String.t()],
98
+
optional(:max_message_size_bytes) => integer()
99
+
}
100
+
101
+
@doc """
102
+
Update filters and options for an active Jetstream connection.
103
+
104
+
Sends an options update message to the Jetstream server over the websocket
105
+
connection. This allows you to dynamically change which collections and DIDs
106
+
you're interested in without reconnecting.
107
+
108
+
## Parameters
109
+
110
+
- `name` - The name of the Jetstream instance (default: `Drinkup.Jetstream`)
111
+
- `opts` - Map with optional fields:
112
+
- `:wanted_collections` - List of collection NSIDs or prefixes (max 100)
113
+
- `:wanted_dids` - List of DIDs to filter (max 10,000)
114
+
- `:max_message_size_bytes` - Maximum message size to receive
115
+
116
+
## Examples
117
+
118
+
# Filter to only posts
119
+
Drinkup.Jetstream.update_options(MyJetstream, %{
120
+
wanted_collections: ["app.bsky.feed.post"]
121
+
})
122
+
123
+
# Filter to specific DIDs
124
+
Drinkup.Jetstream.update_options(MyJetstream, %{
125
+
wanted_dids: ["did:plc:abc123", "did:plc:def456"]
126
+
})
127
+
128
+
# Disable all filters (receive all events)
129
+
Drinkup.Jetstream.update_options(MyJetstream, %{
130
+
wanted_collections: [],
131
+
wanted_dids: []
132
+
})
133
+
134
+
## Return Value
135
+
136
+
Returns `:ok` if the message was sent successfully, or `{:error, reason}` if
137
+
the socket process could not be found or the message could not be sent.
138
+
139
+
Note: The server may reject invalid updates (e.g., too many collections/DIDs).
140
+
Invalid updates will result in the connection being closed by the server.
141
+
"""
142
+
@spec update_options(atom(), update_opts()) :: :ok | {:error, term()}
143
+
def update_options(name \\ Drinkup.Jetstream, opts) when is_map(opts) do
144
+
case find_connection(name) do
145
+
{:ok, {conn, stream}} ->
146
+
message = build_options_update_message(opts)
147
+
:ok = :gun.ws_send(conn, stream, {:text, message})
148
+
149
+
Logger.debug("[Drinkup.Jetstream] Sent options update")
150
+
:ok
151
+
152
+
{:error, reason} ->
153
+
{:error, reason}
154
+
end
155
+
end
156
+
157
+
# Private functions
158
+
159
+
@spec find_connection(atom()) :: {:ok, {pid(), :gun.stream_ref()}} | {:error, :not_connected}
160
+
defp find_connection(name) do
161
+
# Look up the connection details from Registry
162
+
case Registry.lookup(Drinkup.Registry, {name, JetstreamConnection}) do
163
+
[{_socket_pid, {conn, stream}}] ->
164
+
{:ok, {conn, stream}}
165
+
166
+
[] ->
167
+
{:error, :not_connected}
168
+
end
169
+
end
170
+
171
+
@spec build_options_update_message(update_opts()) :: String.t()
172
+
defp build_options_update_message(opts) do
173
+
payload =
174
+
%{}
175
+
|> maybe_add_wanted_collections(Map.get(opts, :wanted_collections))
176
+
|> maybe_add_wanted_dids(Map.get(opts, :wanted_dids))
177
+
|> maybe_add_max_message_size(Map.get(opts, :max_message_size_bytes))
178
+
179
+
message = %{
180
+
"type" => "options_update",
181
+
"payload" => payload
182
+
}
183
+
184
+
Jason.encode!(message)
185
+
end
186
+
187
+
@spec maybe_add_wanted_collections(map(), [String.t()] | nil) :: map()
188
+
defp maybe_add_wanted_collections(payload, nil), do: payload
189
+
190
+
defp maybe_add_wanted_collections(payload, collections) when is_list(collections) do
191
+
Map.put(payload, "wantedCollections", collections)
192
+
end
193
+
194
+
@spec maybe_add_wanted_dids(map(), [String.t()] | nil) :: map()
195
+
defp maybe_add_wanted_dids(payload, nil), do: payload
196
+
197
+
defp maybe_add_wanted_dids(payload, dids) when is_list(dids) do
198
+
Map.put(payload, "wantedDids", dids)
199
+
end
200
+
201
+
@spec maybe_add_max_message_size(map(), integer() | nil) :: map()
202
+
defp maybe_add_max_message_size(payload, nil), do: payload
203
+
204
+
defp maybe_add_max_message_size(payload, max_size) when is_integer(max_size) do
205
+
Map.put(payload, "maxMessageSizeBytes", max_size)
206
+
end
207
+
end
-22
lib/options.ex
-22
lib/options.ex
···
1
-
defmodule Drinkup.Options do
2
-
use TypedStruct
3
-
4
-
@default_host "https://bsky.network"
5
-
6
-
@type options() :: %{
7
-
required(:consumer) => module(),
8
-
optional(:name) => atom(),
9
-
optional(:host) => String.t(),
10
-
optional(:cursor) => pos_integer()
11
-
}
12
-
13
-
typedstruct do
14
-
field :consumer, module(), enforce: true
15
-
field :name, atom(), default: Drinkup
16
-
field :host, String.t(), default: @default_host
17
-
field :cursor, pos_integer() | nil
18
-
end
19
-
20
-
@spec from(options()) :: t()
21
-
def from(%{consumer: _} = options), do: struct(__MODULE__, options)
22
-
end
-85
lib/record_consumer.ex
-85
lib/record_consumer.ex
···
1
-
defmodule Drinkup.RecordConsumer do
2
-
@moduledoc """
3
-
An opinionated consumer of the Firehose that eats consumers
4
-
"""
5
-
6
-
@callback handle_create(any()) :: any()
7
-
@callback handle_update(any()) :: any()
8
-
@callback handle_delete(any()) :: any()
9
-
10
-
defmacro __using__(opts) do
11
-
{collections, _opts} = Keyword.pop(opts, :collections, [])
12
-
13
-
quote location: :keep do
14
-
@behaviour Drinkup.Consumer
15
-
@behaviour Drinkup.RecordConsumer
16
-
17
-
def handle_event(%Drinkup.Event.Commit{} = event) do
18
-
event.ops
19
-
|> Enum.filter(fn %{path: path} ->
20
-
path |> String.split("/") |> Enum.at(0) |> matches_collections?()
21
-
end)
22
-
|> Enum.map(&Drinkup.RecordConsumer.Record.from(&1, event.repo))
23
-
|> Enum.each(&apply(__MODULE__, :"handle_#{&1.action}", [&1]))
24
-
end
25
-
26
-
def handle_event(_event), do: :noop
27
-
28
-
unquote(
29
-
if collections == [] do
30
-
quote do
31
-
def matches_collections?(_type), do: true
32
-
end
33
-
else
34
-
quote do
35
-
def matches_collections?(nil), do: false
36
-
37
-
def matches_collections?(type) when is_binary(type),
38
-
do:
39
-
Enum.any?(unquote(collections), fn
40
-
matcher when is_binary(matcher) -> type == matcher
41
-
matcher -> Regex.match?(matcher, type)
42
-
end)
43
-
end
44
-
end
45
-
)
46
-
47
-
@impl true
48
-
def handle_create(_record), do: nil
49
-
@impl true
50
-
def handle_update(_record), do: nil
51
-
@impl true
52
-
def handle_delete(_record), do: nil
53
-
54
-
defoverridable handle_create: 1, handle_update: 1, handle_delete: 1
55
-
end
56
-
end
57
-
58
-
defmodule Record do
59
-
alias Drinkup.Event.Commit.RepoOp
60
-
use TypedStruct
61
-
62
-
typedstruct do
63
-
field :type, String.t()
64
-
field :rkey, String.t()
65
-
field :did, String.t()
66
-
field :action, :create | :update | :delete
67
-
field :cid, binary() | nil
68
-
field :record, map() | nil
69
-
end
70
-
71
-
@spec from(RepoOp.t(), String.t()) :: t()
72
-
def from(%RepoOp{action: action, path: path, cid: cid, record: record}, did) do
73
-
[type, rkey] = String.split(path, "/")
74
-
75
-
%__MODULE__{
76
-
type: type,
77
-
rkey: rkey,
78
-
did: did,
79
-
action: action,
80
-
cid: cid,
81
-
record: record
82
-
}
83
-
end
84
-
end
85
-
end
+282
-98
lib/socket.ex
+282
-98
lib/socket.ex
···
1
1
defmodule Drinkup.Socket do
2
-
@moduledoc """
3
-
gen_statem process for managing the websocket connection to an ATProto relay.
4
-
"""
2
+
# TODO: talk about how to implment, but that it's for internal use
3
+
@moduledoc false
5
4
6
5
require Logger
7
-
alias Drinkup.{Event, Options}
8
6
9
7
@behaviour :gen_statem
10
-
@timeout :timer.seconds(5)
11
-
# TODO: `flow` determines messages in buffer. Determine ideal value?
12
-
@flow 10
8
+
9
+
@type frame ::
10
+
{:binary, binary()}
11
+
| {:text, String.t()}
12
+
| :close
13
+
| {:close, errno :: integer(), reason :: binary()}
14
+
15
+
@type user_data :: term()
16
+
17
+
@type reconnect_strategy ::
18
+
:exponential
19
+
| {:exponential, max_backoff :: pos_integer()}
20
+
| {:custom, (attempt :: pos_integer() -> delay_ms :: pos_integer())}
21
+
22
+
@type option ::
23
+
{:host, String.t()}
24
+
| {:flow, pos_integer()}
25
+
| {:timeout, pos_integer()}
26
+
| {:tls_opts, keyword()}
27
+
| {:gun_opts, map()}
28
+
| {:reconnect_strategy, reconnect_strategy()}
29
+
| {atom(), term()}
30
+
31
+
@callback init(opts :: keyword()) :: {:ok, user_data()} | {:error, reason :: term()}
32
+
33
+
@callback build_path(data :: user_data()) :: String.t()
34
+
35
+
@callback handle_frame(
36
+
frame :: frame(),
37
+
data :: {user_data(), conn :: pid() | nil, stream :: :gun.stream_ref() | nil}
38
+
) ::
39
+
{:ok, new_data :: user_data()} | :noop | nil | {:error, reason :: term()}
40
+
41
+
@callback handle_connected(data :: {user_data(), conn :: pid(), stream :: :gun.stream_ref()}) ::
42
+
{:ok, new_data :: user_data()}
43
+
44
+
@callback handle_disconnected(
45
+
reason :: term(),
46
+
data :: {user_data(), conn :: pid() | nil, stream :: :gun.stream_ref() | nil}
47
+
) ::
48
+
{:ok, new_data :: user_data()}
49
+
50
+
@optional_callbacks handle_connected: 1, handle_disconnected: 2
51
+
52
+
defstruct [
53
+
:module,
54
+
:user_data,
55
+
:options,
56
+
:conn,
57
+
:stream,
58
+
reconnect_attempts: 0
59
+
]
60
+
61
+
defmacro __using__(_opts) do
62
+
quote do
63
+
@behaviour Drinkup.Socket
64
+
65
+
def start_link(opts, statem_opts \\ [])
66
+
67
+
def start_link(opts, statem_opts) do
68
+
Drinkup.Socket.start_link(__MODULE__, opts, statem_opts)
69
+
end
70
+
71
+
defoverridable start_link: 2
72
+
73
+
def child_spec(opts) do
74
+
%{
75
+
id: __MODULE__,
76
+
start: {__MODULE__, :start_link, [opts, []]},
77
+
type: :worker,
78
+
restart: :permanent,
79
+
shutdown: 500
80
+
}
81
+
end
82
+
83
+
defoverridable child_spec: 1
13
84
14
-
@op_regular 1
15
-
@op_error -1
85
+
@impl true
86
+
def handle_connected({user_data, _conn, _stream}), do: {:ok, user_data}
16
87
17
-
defstruct [:options, :seq, :conn, :stream]
88
+
@impl true
89
+
def handle_disconnected(_reason, {user_data, _conn, _stream}), do: {:ok, user_data}
90
+
91
+
defoverridable handle_connected: 1, handle_disconnected: 2
92
+
end
93
+
end
18
94
19
95
@impl true
20
96
def callback_mode, do: [:state_functions, :state_enter]
21
97
22
-
def child_spec(opts) do
23
-
%{
24
-
id: __MODULE__,
25
-
start: {__MODULE__, :start_link, [opts, []]},
26
-
type: :worker,
27
-
restart: :permanent,
28
-
shutdown: 500
29
-
}
30
-
end
98
+
@doc """
99
+
Start a WebSocket connection process.
31
100
32
-
def start_link(%Options{} = options, statem_opts) do
33
-
:gen_statem.start_link(__MODULE__, options, statem_opts)
101
+
## Parameters
102
+
103
+
* `module` - The module implementing the Drinkup.Socket behaviour
104
+
* `opts` - Keyword list of options (see module documentation)
105
+
* `statem_opts` - Options passed to `:gen_statem.start_link/3`
106
+
"""
107
+
def start_link(module, opts, statem_opts) do
108
+
:gen_statem.start_link(__MODULE__, {module, opts}, statem_opts)
34
109
end
35
110
36
111
@impl true
37
-
def init(%{cursor: seq} = options) do
38
-
data = %__MODULE__{seq: seq, options: options}
39
-
{:ok, :disconnected, data, [{:next_event, :internal, :connect}]}
112
+
def init({module, opts}) do
113
+
case module.init(opts) do
114
+
{:ok, user_data} ->
115
+
options = parse_options(opts)
116
+
117
+
data = %__MODULE__{
118
+
module: module,
119
+
user_data: user_data,
120
+
options: options,
121
+
reconnect_attempts: 0
122
+
}
123
+
124
+
{:ok, :disconnected, data, [{:next_event, :internal, :connect}]}
125
+
126
+
{:error, reason} ->
127
+
{:stop, {:init_failed, reason}}
128
+
end
40
129
end
41
130
42
-
def disconnected(:enter, _from, data) do
43
-
Logger.debug("Initial connection")
44
-
# TODO: differentiate between initial & reconnects, probably stuff to do with seq
45
-
{:next_state, :disconnected, data}
131
+
# :disconnected state - waiting to connect or reconnect
132
+
133
+
def disconnected(:enter, _from, _data) do
134
+
Logger.debug("[Drinkup.Socket] Entering disconnected state")
135
+
:keep_state_and_data
46
136
end
47
137
48
138
def disconnected(:internal, :connect, data) do
49
139
{:next_state, :connecting_http, data}
50
140
end
51
141
142
+
def disconnected(:timeout, :reconnect, data) do
143
+
{:next_state, :connecting_http, data}
144
+
end
145
+
146
+
# :connecting_http state - establishing HTTP connection with TLS
147
+
52
148
def connecting_http(:enter, _from, %{options: options} = data) do
53
-
Logger.debug("Connecting to http")
149
+
Logger.debug("[Drinkup.Socket] Connecting to HTTP")
54
150
55
151
%{host: host, port: port} = URI.new!(options.host)
56
152
57
-
{:ok, conn} =
58
-
:gun.open(:binary.bin_to_list(host), port, %{
59
-
retry: 0,
60
-
protocols: [:http],
61
-
connect_timeout: @timeout,
62
-
domain_lookup_timeout: @timeout,
63
-
tls_handshake_timeout: @timeout,
64
-
tls_opts: [
65
-
verify: :verify_peer,
66
-
cacerts: :certifi.cacerts(),
67
-
depth: 3,
68
-
customize_hostname_check: [
69
-
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
70
-
]
71
-
]
72
-
})
153
+
gun_opts =
154
+
Map.merge(
155
+
%{
156
+
retry: 0,
157
+
protocols: [:http],
158
+
connect_timeout: options.timeout,
159
+
domain_lookup_timeout: options.timeout,
160
+
tls_handshake_timeout: options.timeout,
161
+
tls_opts: options.tls_opts
162
+
},
163
+
options.gun_opts
164
+
)
165
+
166
+
case :gun.open(:binary.bin_to_list(host), port, gun_opts) do
167
+
{:ok, conn} ->
168
+
{:keep_state, %{data | conn: conn}, [{:state_timeout, options.timeout, :connect_timeout}]}
73
169
74
-
{:keep_state, %{data | conn: conn}, [{:state_timeout, @timeout, :connect_timeout}]}
170
+
{:error, reason} ->
171
+
Logger.error("[Drinkup.Socket] Failed to open connection: #{inspect(reason)}")
172
+
{:stop, {:connect_failed, reason}}
173
+
end
75
174
end
76
175
77
176
def connecting_http(:info, {:gun_up, _conn, :http}, data) do
78
177
{:next_state, :connecting_ws, data}
79
178
end
80
179
81
-
def connecting_http(:state_timeout, :connect_timeout, _data) do
82
-
{:stop, :connect_http_timeout}
180
+
def connecting_http(:state_timeout, :connect_timeout, data) do
181
+
Logger.error("[Drinkup.Socket] HTTP connection timeout")
182
+
trigger_reconnect(data)
83
183
end
84
184
85
-
def connecting_ws(:enter, _from, %{conn: conn, seq: seq} = data) do
86
-
Logger.debug("Upgrading connection to websocket")
87
-
path = "/xrpc/com.atproto.sync.subscribeRepos?" <> URI.encode_query(%{cursor: seq})
88
-
stream = :gun.ws_upgrade(conn, path, [], %{flow: @flow})
89
-
{:keep_state, %{data | stream: stream}, [{:state_timeout, @timeout, :upgrade_timeout}]}
185
+
# :connecting_ws state - upgrading to WebSocket
186
+
187
+
def connecting_ws(
188
+
:enter,
189
+
_from,
190
+
%{module: module, user_data: user_data, options: options} = data
191
+
) do
192
+
Logger.debug("[Drinkup.Socket] Upgrading connection to WebSocket")
193
+
194
+
path = module.build_path(user_data)
195
+
stream = :gun.ws_upgrade(data.conn, path, [], %{flow: options.flow})
196
+
197
+
{:keep_state, %{data | stream: stream}, [{:state_timeout, options.timeout, :upgrade_timeout}]}
90
198
end
91
199
92
200
def connecting_ws(:info, {:gun_upgrade, _conn, _stream, ["websocket"], _headers}, data) do
93
201
{:next_state, :connected, data}
94
202
end
95
203
96
-
def connecting_ws(:state_timeout, :upgrade_timeout, _data) do
97
-
{:stop, :connect_ws_timeout}
204
+
def connecting_ws(:info, {:gun_response, _conn, _stream, _fin, status, _headers}, data) do
205
+
Logger.error("[Drinkup.Socket] WebSocket upgrade failed with status: #{status}")
206
+
trigger_reconnect(data)
98
207
end
99
208
100
-
def connected(:enter, _from, _data) do
101
-
Logger.debug("Connected to websocket")
102
-
:keep_state_and_data
209
+
def connecting_ws(:info, {:gun_error, _conn, _stream, reason}, data) do
210
+
Logger.error("[Drinkup.Socket] WebSocket upgrade error: #{inspect(reason)}")
211
+
trigger_reconnect(data)
103
212
end
104
213
105
-
def connected(:info, {:gun_ws, conn, stream, {:binary, frame}}, %{options: options} = data) do
106
-
# TODO: let clients specify a handler for raw* (*decoded) packets to support any atproto subscription
107
-
# Will also need support for JSON frames
108
-
with {:ok, header, next} <- CAR.DagCbor.decode(frame),
109
-
{:ok, payload, _} <- CAR.DagCbor.decode(next),
110
-
{%{"op" => @op_regular, "t" => type}, _} <- {header, payload},
111
-
true <- Event.valid_seq?(data.seq, payload["seq"]) do
112
-
data = %{data | seq: payload["seq"] || data.seq}
113
-
message = Event.from(type, payload)
114
-
:ok = :gun.update_flow(conn, stream, @flow)
214
+
def connecting_ws(:state_timeout, :upgrade_timeout, data) do
215
+
Logger.error("[Drinkup.Socket] WebSocket upgrade timeout")
216
+
trigger_reconnect(data)
217
+
end
218
+
219
+
# :connected state - active WebSocket connection
220
+
221
+
def connected(
222
+
:enter,
223
+
_from,
224
+
%{module: module, user_data: user_data, conn: conn, stream: stream} = data
225
+
) do
226
+
Logger.debug("[Drinkup.Socket] WebSocket connected")
115
227
116
-
case message do
117
-
nil ->
118
-
Logger.warning("Received unrecognised event from firehose: #{inspect({type, payload})}")
228
+
case module.handle_connected({user_data, conn, stream}) do
229
+
{:ok, new_user_data} ->
230
+
{:keep_state, %{data | user_data: new_user_data, reconnect_attempts: 0}}
231
+
232
+
_ ->
233
+
{:keep_state, %{data | reconnect_attempts: 0}}
234
+
end
235
+
end
236
+
237
+
def connected(
238
+
:info,
239
+
{:gun_ws, conn, _stream, frame},
240
+
%{module: module, user_data: user_data, options: options, conn: conn, stream: stream} =
241
+
data
242
+
) do
243
+
result = module.handle_frame(frame, {user_data, conn, stream})
119
244
120
-
message ->
121
-
Event.dispatch(message, options)
122
-
end
245
+
:ok = :gun.update_flow(conn, frame, options.flow)
123
246
124
-
{:keep_state, data}
125
-
else
126
-
false ->
127
-
Logger.error("Got out of sequence or invalid `seq` from Firehose")
128
-
{:keep_state, data}
247
+
case result do
248
+
{:ok, new_user_data} ->
249
+
{:keep_state, %{data | user_data: new_user_data}}
129
250
130
-
{%{"op" => @op_error, "t" => type}, payload} ->
131
-
Logger.error("Got error from Firehose: #{inspect({type, payload})}")
132
-
{:keep_state, data}
251
+
result when result in [:noop, nil] ->
252
+
:keep_state_and_data
133
253
134
254
{:error, reason} ->
135
-
Logger.warning("Failed to decode frame from Firehose: #{inspect(reason)}")
136
-
{:keep_state, data}
255
+
Logger.error("[Drinkup.Socket] Frame handler error: #{inspect(reason)}")
256
+
:keep_state_and_data
137
257
end
138
258
end
139
259
140
-
def connected(:info, {:gun_ws, _conn, _stream, :close}, _data) do
141
-
Logger.info("Websocket closed, reason unknown")
142
-
{:keep_state_and_data, [{:next_event, :internal, :reconnect}]}
260
+
def connected(:info, {:gun_ws, _conn, _stream, :close}, data) do
261
+
Logger.info("[Drinkup.Socket] WebSocket closed by remote")
262
+
trigger_reconnect(data, :remote_close)
143
263
end
144
264
145
-
def connected(:info, {:gun_ws, _conn, _stream, {:close, errno, reason}}, _data) do
146
-
Logger.info("Websocket closed, errno: #{errno}, reason: #{inspect(reason)}")
147
-
{:keep_state_and_data, [{:next_event, :internal, :reconnect}]}
265
+
def connected(:info, {:gun_ws, _conn, _stream, {:close, errno, reason}}, data) do
266
+
Logger.info("[Drinkup.Socket] WebSocket closed: #{errno} - #{inspect(reason)}")
267
+
trigger_reconnect(data, {:remote_close, errno, reason})
148
268
end
149
269
150
270
def connected(:info, {:gun_down, old_conn, _proto, _reason, _killed_streams}, %{conn: new_conn})
151
271
when old_conn != new_conn do
152
-
Logger.debug("Ignoring received :gun_down for a previous connection.")
272
+
Logger.debug("[Drinkup.Socket] Ignoring :gun_down for old connection")
153
273
:keep_state_and_data
154
274
end
155
275
156
-
def connected(:info, {:gun_down, _conn, _proto, _reason, _killed_streams}, _data) do
157
-
Logger.info("Websocket connection killed. Attempting to reconnect")
158
-
{:keep_state_and_data, [{:next_event, :internal, :reconnect}]}
276
+
def connected(:info, {:gun_down, _conn, _proto, reason, _killed_streams}, data) do
277
+
Logger.info("[Drinkup.Socket] Connection down: #{inspect(reason)}")
278
+
trigger_reconnect(data, {:connection_down, reason})
159
279
end
160
280
161
-
def connected(:internal, :reconnect, %{conn: conn} = data) do
281
+
def connected(
282
+
:internal,
283
+
:reconnect,
284
+
%{conn: conn, options: options, reconnect_attempts: attempts} = data
285
+
) do
162
286
:ok = :gun.close(conn)
163
287
:ok = :gun.flush(conn)
164
288
165
-
# TODO: reconnect backoff
166
-
{:next_state, :disconnected, %{data | conn: nil, stream: nil},
167
-
[{:next_event, :internal, :connect}]}
289
+
backoff = calculate_backoff(attempts, options.reconnect_strategy)
290
+
291
+
Logger.info("[Drinkup.Socket] Reconnecting in #{backoff}ms (attempt #{attempts + 1})")
292
+
293
+
{:next_state, :disconnected,
294
+
%{data | conn: nil, stream: nil, reconnect_attempts: attempts + 1},
295
+
[{{:timeout, :reconnect}, backoff, :reconnect}]}
296
+
end
297
+
298
+
# Helper functions
299
+
300
+
defp trigger_reconnect(data, reason \\ :unknown) do
301
+
%{module: module, user_data: user_data, conn: conn, stream: stream} = data
302
+
303
+
case module.handle_disconnected(reason, {user_data, conn, stream}) do
304
+
{:ok, new_user_data} ->
305
+
{:keep_state, %{data | user_data: new_user_data}, [{:next_event, :internal, :reconnect}]}
306
+
307
+
_ ->
308
+
{:keep_state_and_data, [{:next_event, :internal, :reconnect}]}
309
+
end
310
+
end
311
+
312
+
defp parse_options(opts) do
313
+
%{
314
+
host: Keyword.fetch!(opts, :host),
315
+
flow: Keyword.get(opts, :flow, 10),
316
+
timeout: Keyword.get(opts, :timeout, :timer.seconds(5)),
317
+
tls_opts: Keyword.get(opts, :tls_opts, default_tls_opts()),
318
+
gun_opts: Keyword.get(opts, :gun_opts, %{}),
319
+
reconnect_strategy: Keyword.get(opts, :reconnect_strategy, :exponential)
320
+
}
321
+
end
322
+
323
+
defp default_tls_opts do
324
+
[
325
+
verify: :verify_peer,
326
+
cacerts: :certifi.cacerts(),
327
+
depth: 3,
328
+
customize_hostname_check: [
329
+
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
330
+
]
331
+
]
332
+
end
333
+
334
+
defp calculate_backoff(attempt, strategy) do
335
+
case strategy do
336
+
:exponential ->
337
+
exponential_backoff(attempt, :timer.seconds(60))
338
+
339
+
{:exponential, max_backoff} ->
340
+
exponential_backoff(attempt, max_backoff)
341
+
342
+
{:custom, func} when is_function(func, 1) ->
343
+
func.(attempt)
344
+
end
345
+
end
346
+
347
+
defp exponential_backoff(attempt, max_backoff) do
348
+
base = :timer.seconds(1)
349
+
delay = min(base * :math.pow(2, attempt), max_backoff)
350
+
jitter = :rand.uniform(trunc(delay * 0.1))
351
+
trunc(delay) + jitter
168
352
end
169
353
end
+46
lib/tap/consumer.ex
+46
lib/tap/consumer.ex
···
1
+
defmodule Drinkup.Tap.Consumer do
2
+
@moduledoc """
3
+
Consumer behaviour for handling Tap events.
4
+
5
+
Implement this behaviour to process events from a Tap indexer/backfill service.
6
+
Events are dispatched asynchronously via `Task.Supervisor` and acknowledged
7
+
to Tap based on the return value of `handle_event/1`.
8
+
9
+
## Event Acknowledgment
10
+
11
+
By default, events are acknowledged to Tap based on your return value:
12
+
13
+
- `:ok`, `{:ok, any()}`, or `nil` → Success, event is acked to Tap
14
+
- `{:error, reason}` → Failure, event is NOT acked (Tap will retry after timeout)
15
+
- Exception raised → Failure, event is NOT acked (Tap will retry after timeout)
16
+
17
+
Any other value will log a warning and acknowledge the event anyway.
18
+
19
+
If you set `disable_acks: true` in your Tap options, no acks are sent regardless
20
+
of the return value. This matches Tap's `TAP_DISABLE_ACKS` environment variable.
21
+
22
+
## Example
23
+
24
+
defmodule MyTapConsumer do
25
+
@behaviour Drinkup.Tap.Consumer
26
+
27
+
def handle_event(%Drinkup.Tap.Event.Record{action: :create} = record) do
28
+
# Handle new record creation
29
+
case save_to_database(record) do
30
+
:ok -> :ok # Success - event will be acked
31
+
{:error, reason} -> {:error, reason} # Failure - Tap will retry
32
+
end
33
+
end
34
+
35
+
def handle_event(%Drinkup.Tap.Event.Identity{} = identity) do
36
+
# Handle identity changes
37
+
update_identity(identity)
38
+
:ok # Success - event will be acked
39
+
end
40
+
end
41
+
"""
42
+
43
+
alias Drinkup.Tap.Event
44
+
45
+
@callback handle_event(Event.Record.t() | Event.Identity.t()) :: any()
46
+
end
+39
lib/tap/event/identity.ex
+39
lib/tap/event/identity.ex
···
1
+
defmodule Drinkup.Tap.Event.Identity do
2
+
@moduledoc """
3
+
Struct for identity events from Tap.
4
+
5
+
Represents handle or status changes for a DID.
6
+
"""
7
+
8
+
use TypedStruct
9
+
10
+
typedstruct enforce: true do
11
+
field :id, integer()
12
+
field :did, String.t()
13
+
field :handle, String.t() | nil
14
+
field :is_active, boolean()
15
+
field :status, String.t()
16
+
end
17
+
18
+
@spec from(map()) :: t()
19
+
def from(%{
20
+
"id" => id,
21
+
"type" => "identity",
22
+
"identity" =>
23
+
%{
24
+
"did" => did,
25
+
"is_active" => is_active,
26
+
"status" => status
27
+
} = identity_data
28
+
}) do
29
+
handle = Map.get(identity_data, "handle")
30
+
31
+
%__MODULE__{
32
+
id: id,
33
+
did: did,
34
+
handle: handle,
35
+
is_active: is_active,
36
+
status: status
37
+
}
38
+
end
39
+
end
+58
lib/tap/event/record.ex
+58
lib/tap/event/record.ex
···
1
+
defmodule Drinkup.Tap.Event.Record do
2
+
@moduledoc """
3
+
Struct for record events from Tap.
4
+
5
+
Represents create, update, or delete operations on records in the repository.
6
+
"""
7
+
8
+
use TypedStruct
9
+
10
+
typedstruct enforce: true do
11
+
@type action() :: :create | :update | :delete
12
+
13
+
field :id, integer()
14
+
field :live, boolean()
15
+
field :rev, String.t()
16
+
field :did, String.t()
17
+
field :collection, String.t()
18
+
field :rkey, String.t()
19
+
field :action, action()
20
+
field :cid, String.t() | nil
21
+
field :record, map() | nil
22
+
end
23
+
24
+
@spec from(map()) :: t()
25
+
def from(%{
26
+
"id" => id,
27
+
"type" => "record",
28
+
"record" =>
29
+
%{
30
+
"live" => live,
31
+
"rev" => rev,
32
+
"did" => did,
33
+
"collection" => collection,
34
+
"rkey" => rkey,
35
+
"action" => action
36
+
} = record_data
37
+
}) do
38
+
cid = Map.get(record_data, "cid")
39
+
record = Map.get(record_data, "record")
40
+
41
+
%__MODULE__{
42
+
id: id,
43
+
live: live,
44
+
rev: rev,
45
+
did: did,
46
+
collection: collection,
47
+
rkey: rkey,
48
+
action: parse_action(action),
49
+
cid: cid,
50
+
record: record
51
+
}
52
+
end
53
+
54
+
@spec parse_action(String.t()) :: action()
55
+
defp parse_action("create"), do: :create
56
+
defp parse_action("update"), do: :update
57
+
defp parse_action("delete"), do: :delete
58
+
end
+105
lib/tap/event.ex
+105
lib/tap/event.ex
···
1
+
defmodule Drinkup.Tap.Event do
2
+
@moduledoc """
3
+
Event handling and dispatch for Tap events.
4
+
5
+
Parses incoming JSON events from Tap and dispatches them to the configured
6
+
consumer via Task.Supervisor. After successful processing, sends an ack
7
+
message back to the socket.
8
+
"""
9
+
10
+
require Logger
11
+
alias Drinkup.Tap.{Event, Options}
12
+
13
+
@type t() :: Event.Record.t() | Event.Identity.t()
14
+
15
+
@doc """
16
+
Parse a JSON map into an event struct.
17
+
18
+
Returns the appropriate event struct based on the "type" field.
19
+
"""
20
+
@spec from(map()) :: t() | nil
21
+
def from(%{"type" => "record"} = payload), do: Event.Record.from(payload)
22
+
def from(%{"type" => "identity"} = payload), do: Event.Identity.from(payload)
23
+
def from(_payload), do: nil
24
+
25
+
@doc """
26
+
Dispatch an event to the consumer via Task.Supervisor.
27
+
28
+
Spawns a task that:
29
+
1. Processes the event via the consumer's handle_event/1 callback
30
+
2. Sends an ack to Tap if acks are enabled and the consumer returns :ok, {:ok, _}, or nil
31
+
3. Does not ack if the consumer returns an error-like value or raises an exception
32
+
33
+
Consumer return value semantics (when acks are enabled):
34
+
- `:ok` or `{:ok, any()}` or `nil` -> Success, send ack
35
+
- `{:error, _}` or any error-like tuple -> Failure, don't ack (Tap will retry)
36
+
- Exception raised -> Failure, don't ack (Tap will retry)
37
+
38
+
If `disable_acks: true` is set in options, no acks are sent regardless of
39
+
consumer return value.
40
+
"""
41
+
@spec dispatch(t(), Options.t(), pid(), :gun.stream_ref()) :: :ok
42
+
def dispatch(
43
+
event,
44
+
%Options{consumer: consumer, name: name, disable_acks: disable_acks},
45
+
conn,
46
+
stream
47
+
) do
48
+
supervisor_name = {:via, Registry, {Drinkup.Registry, {name, TapTasks}}}
49
+
event_id = get_event_id(event)
50
+
51
+
{:ok, _pid} =
52
+
Task.Supervisor.start_child(supervisor_name, fn ->
53
+
try do
54
+
result = consumer.handle_event(event)
55
+
56
+
unless disable_acks do
57
+
case result do
58
+
:ok ->
59
+
send_ack(conn, stream, event_id)
60
+
61
+
{:ok, _} ->
62
+
send_ack(conn, stream, event_id)
63
+
64
+
nil ->
65
+
send_ack(conn, stream, event_id)
66
+
67
+
:error ->
68
+
Logger.error("Consumer returned error for event #{event_id}, not acking.")
69
+
70
+
{:error, reason} ->
71
+
Logger.error(
72
+
"Consumer returned error for event #{event_id}, not acking: #{inspect(reason)}"
73
+
)
74
+
75
+
_ ->
76
+
Logger.warning(
77
+
"Consumer returned unexpected value for event #{event_id}, acking anyway: #{inspect(result)}"
78
+
)
79
+
80
+
send_ack(conn, stream, event_id)
81
+
end
82
+
end
83
+
rescue
84
+
e ->
85
+
Logger.error(
86
+
"Error in Tap event handler (event #{event_id}), not acking: #{Exception.format(:error, e, __STACKTRACE__)}"
87
+
)
88
+
end
89
+
end)
90
+
91
+
:ok
92
+
end
93
+
94
+
@spec send_ack(pid(), :gun.stream_ref(), integer()) :: :ok
95
+
defp send_ack(conn, stream, event_id) do
96
+
ack_message = Jason.encode!(%{type: "ack", id: event_id})
97
+
98
+
:ok = :gun.ws_send(conn, stream, {:text, ack_message})
99
+
Logger.debug("[Drinkup.Tap] Acked event #{event_id}")
100
+
end
101
+
102
+
@spec get_event_id(t()) :: integer()
103
+
defp get_event_id(%Event.Record{id: id}), do: id
104
+
defp get_event_id(%Event.Identity{id: id}), do: id
105
+
end
+90
lib/tap/options.ex
+90
lib/tap/options.ex
···
1
+
defmodule Drinkup.Tap.Options do
2
+
@moduledoc """
3
+
Configuration options for Tap indexer/backfill service connection.
4
+
5
+
This module defines the configuration structure for connecting to and
6
+
interacting with a Tap service. Tap simplifies AT Protocol sync by handling
7
+
firehose connections, verification, backfill, and filtering server-side.
8
+
9
+
## Options
10
+
11
+
- `:consumer` (required) - Module implementing `Drinkup.Tap.Consumer` behaviour
12
+
- `:name` - Unique name for this Tap instance in the supervision tree (default: `Drinkup.Tap`)
13
+
- `:host` - Tap service URL (default: `"http://localhost:2480"`)
14
+
- `:admin_password` - Optional password for authenticated Tap instances
15
+
- `:disable_acks` - Disable event acknowledgments (default: `false`)
16
+
17
+
## Example
18
+
19
+
%{
20
+
consumer: MyTapConsumer,
21
+
name: MyTap,
22
+
host: "http://localhost:2480",
23
+
admin_password: "secret",
24
+
disable_acks: false
25
+
}
26
+
"""
27
+
28
+
use TypedStruct
29
+
30
+
@default_host "http://localhost:2480"
31
+
32
+
@typedoc """
33
+
Map of configuration options accepted by `Drinkup.Tap.child_spec/1`.
34
+
"""
35
+
@type options() :: %{
36
+
required(:consumer) => consumer(),
37
+
optional(:name) => name(),
38
+
optional(:host) => host(),
39
+
optional(:admin_password) => admin_password(),
40
+
optional(:disable_acks) => disable_acks()
41
+
}
42
+
43
+
@typedoc """
44
+
Module implementing the `Drinkup.Tap.Consumer` behaviour.
45
+
"""
46
+
@type consumer() :: module()
47
+
48
+
@typedoc """
49
+
Unique identifier for this Tap instance in the supervision tree.
50
+
51
+
Used for Registry lookups and naming child processes.
52
+
"""
53
+
@type name() :: atom()
54
+
55
+
@typedoc """
56
+
HTTP/HTTPS URL of the Tap service.
57
+
58
+
Defaults to `"http://localhost:2480"` which is Tap's default bind address.
59
+
"""
60
+
@type host() :: String.t()
61
+
62
+
@typedoc """
63
+
Optional password for HTTP Basic authentication.
64
+
65
+
Required when connecting to a Tap service configured with `TAP_ADMIN_PASSWORD`.
66
+
The password is sent as `Basic admin:<password>` in the Authorization header.
67
+
"""
68
+
@type admin_password() :: String.t() | nil
69
+
70
+
@typedoc """
71
+
Whether to disable event acknowledgments.
72
+
73
+
When `true`, events are not acknowledged to Tap regardless of consumer
74
+
return values. This matches Tap's `TAP_DISABLE_ACKS` environment variable.
75
+
76
+
Defaults to `false` (acknowledgments enabled).
77
+
"""
78
+
@type disable_acks() :: boolean()
79
+
80
+
typedstruct do
81
+
field :consumer, consumer(), enforce: true
82
+
field :name, name(), default: Drinkup.Tap
83
+
field :host, host(), default: @default_host
84
+
field :admin_password, admin_password()
85
+
field :disable_acks, disable_acks(), default: false
86
+
end
87
+
88
+
@spec from(options()) :: t()
89
+
def from(%{consumer: _} = options), do: struct(__MODULE__, options)
90
+
end
+100
lib/tap/socket.ex
+100
lib/tap/socket.ex
···
1
+
defmodule Drinkup.Tap.Socket do
2
+
@moduledoc """
3
+
WebSocket connection handler for Tap indexer/backfill service.
4
+
5
+
Implements the Drinkup.Socket behaviour to manage connections to a Tap service,
6
+
handling JSON-encoded events and dispatching them to the configured consumer.
7
+
8
+
Events are acknowledged after successful processing based on the consumer's
9
+
return value:
10
+
- `:ok`, `{:ok, any()}`, or `nil` → Success, ack sent to Tap
11
+
- `{:error, reason}` → Failure, no ack (Tap will retry after timeout)
12
+
- Exception raised → Failure, no ack (Tap will retry after timeout)
13
+
"""
14
+
15
+
use Drinkup.Socket
16
+
17
+
require Logger
18
+
alias Drinkup.Tap.{Event, Options}
19
+
20
+
@impl true
21
+
def init(opts) do
22
+
options = Keyword.fetch!(opts, :options)
23
+
{:ok, %{options: options, host: options.host}}
24
+
end
25
+
26
+
def start_link(%Options{} = options, statem_opts) do
27
+
socket_opts = build_socket_opts(options)
28
+
Drinkup.Socket.start_link(__MODULE__, socket_opts, statem_opts)
29
+
end
30
+
31
+
@impl true
32
+
def build_path(_data) do
33
+
"/channel"
34
+
end
35
+
36
+
@impl true
37
+
def handle_frame({:text, json}, {%{options: options} = data, conn, stream}) do
38
+
case Jason.decode(json) do
39
+
{:ok, payload} ->
40
+
case Event.from(payload) do
41
+
nil ->
42
+
Logger.warning("Received unrecognized event from Tap: #{inspect(payload)}")
43
+
:noop
44
+
45
+
event ->
46
+
Event.dispatch(event, options, conn, stream)
47
+
{:ok, data}
48
+
end
49
+
50
+
{:error, reason} ->
51
+
Logger.error("Failed to decode JSON from Tap: #{inspect(reason)}")
52
+
:noop
53
+
end
54
+
end
55
+
56
+
@impl true
57
+
def handle_frame({:binary, _binary}, _data) do
58
+
Logger.warning("Received unexpected binary frame from Tap")
59
+
:noop
60
+
end
61
+
62
+
@impl true
63
+
def handle_frame(:close, _data) do
64
+
Logger.info("Websocket closed, reason unknown")
65
+
nil
66
+
end
67
+
68
+
@impl true
69
+
def handle_frame({:close, errno, reason}, _data) do
70
+
Logger.info("Websocket closed, errno: #{errno}, reason: #{inspect(reason)}")
71
+
nil
72
+
end
73
+
74
+
defp build_socket_opts(%Options{host: host, admin_password: admin_password} = options) do
75
+
base_opts = [
76
+
host: host,
77
+
options: options
78
+
]
79
+
80
+
if admin_password do
81
+
auth_header = build_auth_header(admin_password)
82
+
83
+
gun_opts = %{
84
+
ws_opts: %{
85
+
headers: [{"authorization", auth_header}]
86
+
}
87
+
}
88
+
89
+
Keyword.put(base_opts, :gun_opts, gun_opts)
90
+
else
91
+
base_opts
92
+
end
93
+
end
94
+
95
+
@spec build_auth_header(String.t()) :: String.t()
96
+
defp build_auth_header(password) do
97
+
credentials = "admin:#{password}"
98
+
"Basic #{Base.encode64(credentials)}"
99
+
end
100
+
end
+249
lib/tap.ex
+249
lib/tap.ex
···
1
+
defmodule Drinkup.Tap do
2
+
@moduledoc """
3
+
Supervisor and HTTP API for Tap indexer/backfill service.
4
+
5
+
Tap simplifies AT sync by handling the firehose connection, verification,
6
+
backfill, and filtering. Your application connects to a Tap service and
7
+
receives simple JSON events for only the repos and collections you care about.
8
+
9
+
## Usage
10
+
11
+
Add Tap to your supervision tree:
12
+
13
+
children = [
14
+
{Drinkup.Tap, %{
15
+
consumer: MyTapConsumer,
16
+
name: MyTap,
17
+
host: "http://localhost:2480",
18
+
admin_password: "secret" # optional
19
+
}}
20
+
]
21
+
22
+
Then interact with the Tap HTTP API:
23
+
24
+
# Add repos to track (triggers backfill)
25
+
Drinkup.Tap.add_repos(MyTap, ["did:plc:abc123"])
26
+
27
+
# Get stats
28
+
{:ok, count} = Drinkup.Tap.get_repo_count(MyTap)
29
+
30
+
## Configuration
31
+
32
+
Tap itself is configured via environment variables. See the Tap documentation
33
+
for details on configuring collection filters, signal collections, and other
34
+
operational settings:
35
+
https://github.com/bluesky-social/indigo/blob/main/cmd/tap/README.md
36
+
"""
37
+
38
+
use Supervisor
39
+
alias Drinkup.Tap.Options
40
+
41
+
@dialyzer nowarn_function: {:init, 1}
42
+
@impl true
43
+
def init({%Options{name: name} = drinkup_options, supervisor_options}) do
44
+
# Register options in Registry for HTTP API access
45
+
Registry.register(Drinkup.Registry, {name, TapOptions}, drinkup_options)
46
+
47
+
children = [
48
+
{Task.Supervisor, name: {:via, Registry, {Drinkup.Registry, {name, TapTasks}}}},
49
+
{Drinkup.Tap.Socket, drinkup_options}
50
+
]
51
+
52
+
Supervisor.start_link(
53
+
children,
54
+
supervisor_options ++ [name: {:via, Registry, {Drinkup.Registry, {name, TapSupervisor}}}]
55
+
)
56
+
end
57
+
58
+
@spec child_spec(Options.options()) :: Supervisor.child_spec()
59
+
def child_spec(%{} = options), do: child_spec({options, [strategy: :one_for_one]})
60
+
61
+
@spec child_spec({Options.options(), Keyword.t()}) :: Supervisor.child_spec()
62
+
def child_spec({drinkup_options, supervisor_options}) do
63
+
%{
64
+
id: Map.get(drinkup_options, :name, __MODULE__),
65
+
start: {__MODULE__, :init, [{Options.from(drinkup_options), supervisor_options}]},
66
+
type: :supervisor,
67
+
restart: :permanent,
68
+
shutdown: 500
69
+
}
70
+
end
71
+
72
+
# HTTP API Functions
73
+
74
+
@doc """
75
+
Add DIDs to track.
76
+
77
+
Triggers backfill for the specified DIDs. Historical events will be fetched
78
+
from each repo's PDS, followed by live events from the firehose.
79
+
"""
80
+
@spec add_repos(atom(), [String.t()]) :: {:ok, term()} | {:error, term()}
81
+
def add_repos(name \\ Drinkup.Tap, dids) when is_list(dids) do
82
+
with {:ok, options} <- get_options(name),
83
+
{:ok, response} <- make_request(options, :post, "/repos/add", %{dids: dids}) do
84
+
{:ok, response}
85
+
end
86
+
end
87
+
88
+
@doc """
89
+
Remove DIDs from tracking.
90
+
91
+
Stops syncing the specified repos and deletes tracked repo metadata. Does not
92
+
delete buffered events in the outbox.
93
+
"""
94
+
@spec remove_repos(atom(), [String.t()]) :: {:ok, term()} | {:error, term()}
95
+
def remove_repos(name \\ Drinkup.Tap, dids) when is_list(dids) do
96
+
with {:ok, options} <- get_options(name),
97
+
{:ok, response} <- make_request(options, :post, "/repos/remove", %{dids: dids}) do
98
+
{:ok, response}
99
+
end
100
+
end
101
+
102
+
@doc """
103
+
Resolve a DID to its DID document.
104
+
"""
105
+
@spec resolve_did(atom(), String.t()) :: {:ok, term()} | {:error, term()}
106
+
def resolve_did(name \\ Drinkup.Tap, did) when is_binary(did) do
107
+
with {:ok, options} <- get_options(name),
108
+
{:ok, response} <- make_request(options, :get, "/resolve/#{did}") do
109
+
{:ok, response}
110
+
end
111
+
end
112
+
113
+
@doc """
114
+
Get info about a tracked repo.
115
+
116
+
Returns repo state, repo rev, record count, error info, and retry count.
117
+
"""
118
+
@spec get_repo_info(atom(), String.t()) :: {:ok, term()} | {:error, term()}
119
+
def get_repo_info(name \\ Drinkup.Tap, did) when is_binary(did) do
120
+
with {:ok, options} <- get_options(name),
121
+
{:ok, response} <- make_request(options, :get, "/info/#{did}") do
122
+
{:ok, response}
123
+
end
124
+
end
125
+
126
+
@doc """
127
+
Get the total number of tracked repos.
128
+
"""
129
+
@spec get_repo_count(atom()) :: {:ok, integer()} | {:error, term()}
130
+
def get_repo_count(name \\ Drinkup.Tap) do
131
+
with {:ok, options} <- get_options(name),
132
+
{:ok, response} <- make_request(options, :get, "/stats/repo-count") do
133
+
{:ok, response}
134
+
end
135
+
end
136
+
137
+
@doc """
138
+
Get the total number of tracked records.
139
+
"""
140
+
@spec get_record_count(atom()) :: {:ok, integer()} | {:error, term()}
141
+
def get_record_count(name \\ Drinkup.Tap) do
142
+
with {:ok, options} <- get_options(name),
143
+
{:ok, response} <- make_request(options, :get, "/stats/record-count") do
144
+
{:ok, response}
145
+
end
146
+
end
147
+
148
+
@doc """
149
+
Get the number of events in the outbox buffer.
150
+
"""
151
+
@spec get_outbox_buffer(atom()) :: {:ok, integer()} | {:error, term()}
152
+
def get_outbox_buffer(name \\ Drinkup.Tap) do
153
+
with {:ok, options} <- get_options(name),
154
+
{:ok, response} <- make_request(options, :get, "/stats/outbox-buffer") do
155
+
{:ok, response}
156
+
end
157
+
end
158
+
159
+
@doc """
160
+
Get the number of events in the resync buffer.
161
+
"""
162
+
@spec get_resync_buffer(atom()) :: {:ok, integer()} | {:error, term()}
163
+
def get_resync_buffer(name \\ Drinkup.Tap) do
164
+
with {:ok, options} <- get_options(name),
165
+
{:ok, response} <- make_request(options, :get, "/stats/resync-buffer") do
166
+
{:ok, response}
167
+
end
168
+
end
169
+
170
+
@doc """
171
+
Get current firehose and list repos cursors.
172
+
"""
173
+
@spec get_cursors(atom()) :: {:ok, map()} | {:error, term()}
174
+
def get_cursors(name \\ Drinkup.Tap) do
175
+
with {:ok, options} <- get_options(name),
176
+
{:ok, response} <- make_request(options, :get, "/stats/cursors") do
177
+
{:ok, response}
178
+
end
179
+
end
180
+
181
+
@doc """
182
+
Check Tap health status.
183
+
184
+
Returns `{:ok, %{"status" => "ok"}}` if healthy.
185
+
"""
186
+
@spec health(atom()) :: {:ok, map()} | {:error, term()}
187
+
def health(name \\ Drinkup.Tap) do
188
+
with {:ok, options} <- get_options(name),
189
+
{:ok, response} <- make_request(options, :get, "/health") do
190
+
{:ok, response}
191
+
end
192
+
end
193
+
194
+
# Private Functions
195
+
196
+
@spec get_options(atom()) :: {:ok, Options.t()} | {:error, :not_found}
197
+
defp get_options(name) do
198
+
case Registry.lookup(Drinkup.Registry, {name, TapOptions}) do
199
+
[{_pid, options}] -> {:ok, options}
200
+
[] -> {:error, :not_found}
201
+
end
202
+
end
203
+
204
+
@spec make_request(Options.t(), atom(), String.t(), map() | nil) ::
205
+
{:ok, term()} | {:error, term()}
206
+
defp make_request(options, method, path, body \\ nil) do
207
+
url = build_url(options.host, path)
208
+
headers = build_headers(options.admin_password)
209
+
210
+
request_opts = [
211
+
method: method,
212
+
url: url,
213
+
headers: headers
214
+
]
215
+
216
+
request_opts =
217
+
if body do
218
+
Keyword.merge(request_opts, json: body)
219
+
else
220
+
request_opts
221
+
end
222
+
223
+
case Req.request(request_opts) do
224
+
{:ok, %{status: status, body: body}} when status in 200..299 ->
225
+
{:ok, body}
226
+
227
+
{:ok, %{status: status, body: body}} ->
228
+
{:error, {:http_error, status, body}}
229
+
230
+
{:error, reason} ->
231
+
{:error, reason}
232
+
end
233
+
end
234
+
235
+
@spec build_url(String.t(), String.t()) :: String.t()
236
+
defp build_url(host, path) do
237
+
host = String.trim_trailing(host, "/")
238
+
"#{host}#{path}"
239
+
end
240
+
241
+
@spec build_headers(String.t() | nil) :: list()
242
+
defp build_headers(nil), do: []
243
+
244
+
defp build_headers(admin_password) do
245
+
credentials = "admin:#{admin_password}"
246
+
auth_header = "Basic #{Base.encode64(credentials)}"
247
+
[{"authorization", auth_header}]
248
+
end
249
+
end
+5
-1
mix.exs
+5
-1
mix.exs
···
34
34
{:certifi, "~> 2.15"},
35
35
{:credo, "~> 1.7", only: [:dev, :test], runtime: false},
36
36
{:ex_doc, "~> 0.34", only: :dev, runtime: false},
37
+
{:ezstd, "~> 1.1"},
37
38
{:gun, "~> 2.2"},
38
-
{:typedstruct, "~> 0.5"}
39
+
{:typedstruct, "~> 0.5"},
40
+
{:jason, "~> 1.4"},
41
+
{:req, "~> 0.5.0"},
42
+
{:atex, "~> 0.7"}
39
43
]
40
44
end
41
45
+27
-6
mix.lock
+27
-6
mix.lock
···
1
1
%{
2
+
"atex": {:hex, :atex, "0.7.0", "23baa616d584ef2cdd2c444b838b672d4472cdae6894fc98dbcb8e1d4b3dd210", [:mix], [{:con_cache, "~> 1.1", [hex: :con_cache, repo: "hexpm", optional: false]}, {:ex_cldr, "~> 2.42", [hex: :ex_cldr, repo: "hexpm", optional: false]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}, {:jose, "~> 1.11", [hex: :jose, repo: "hexpm", optional: false]}, {:multiformats_ex, "~> 0.2", [hex: :multiformats_ex, repo: "hexpm", optional: false]}, {:mutex, "~> 3.0", [hex: :mutex, repo: "hexpm", optional: false]}, {:peri, "~> 0.6", [hex: :peri, repo: "hexpm", optional: false]}, {:plug, "~> 1.18", [hex: :plug, repo: "hexpm", optional: false]}, {:recase, "~> 0.5", [hex: :recase, repo: "hexpm", optional: false]}, {:req, "~> 0.5", [hex: :req, repo: "hexpm", optional: false]}, {:typedstruct, "~> 0.5", [hex: :typedstruct, repo: "hexpm", optional: false]}], "hexpm", "dfb5ced5259658ed6881add0e304b726cd281d7ae030813f7c4fe1e5fa8b35ef"},
2
3
"bunt": {:hex, :bunt, "1.0.0", "081c2c665f086849e6d57900292b3a161727ab40431219529f13c4ddcf3e7a44", [:mix], [], "hexpm", "dc5f86aa08a5f6fa6b8096f0735c4e76d54ae5c9fa2c143e5a1fc7c1cd9bb6b5"},
3
4
"car": {:hex, :car, "0.1.1", "a5bc4c5c1be96eab437634b3c0ccad1fe17b5e3d68c22a4031241ae1345aebd4", [:mix], [{:cbor, "~> 1.0.0", [hex: :cbor, repo: "hexpm", optional: false]}, {:typedstruct, "~> 0.5", [hex: :typedstruct, repo: "hexpm", optional: false]}, {:varint, "~> 1.4", [hex: :varint, repo: "hexpm", optional: false]}], "hexpm", "f895dda8123d04dd336db5a2bf0d0b47f4559cd5383f83fcca0700c1b45bfb6a"},
4
5
"cbor": {:hex, :cbor, "1.0.1", "39511158e8ea5a57c1fcb9639aaa7efde67129678fee49ebbda780f6f24959b0", [:mix], [], "hexpm", "5431acbe7a7908f17f6a9cd43311002836a34a8ab01876918d8cfb709cd8b6a2"},
5
-
"certifi": {:hex, :certifi, "2.15.0", "0e6e882fcdaaa0a5a9f2b3db55b1394dba07e8d6d9bcad08318fb604c6839712", [:rebar3], [], "hexpm", "b147ed22ce71d72eafdad94f055165c1c182f61a2ff49df28bcc71d1d5b94a60"},
6
-
"cowlib": {:hex, :cowlib, "2.15.0", "3c97a318a933962d1c12b96ab7c1d728267d2c523c25a5b57b0f93392b6e9e25", [:make, :rebar3], [], "hexpm", "4f00c879a64b4fe7c8fcb42a4281925e9ffdb928820b03c3ad325a617e857532"},
7
-
"credo": {:hex, :credo, "1.7.12", "9e3c20463de4b5f3f23721527fcaf16722ec815e70ff6c60b86412c695d426c1", [:mix], [{:bunt, "~> 0.2.1 or ~> 1.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2 or ~> 1.0", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "8493d45c656c5427d9c729235b99d498bd133421f3e0a683e5c1b561471291e5"},
6
+
"certifi": {:hex, :certifi, "2.16.0", "a4edfc1d2da3424d478a3271133bf28e0ec5e6fd8c009aab5a4ae980cb165ce9", [:rebar3], [], "hexpm", "8a64f6669d85e9cc0e5086fcf29a5b13de57a13efa23d3582874b9a19303f184"},
7
+
"cldr_utils": {:hex, :cldr_utils, "2.29.1", "11ff0a50a36a7e5f3bd9fc2fb8486a4c1bcca3081d9c080bf9e48fe0e6742e2d", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:certifi, "~> 2.5", [hex: :certifi, repo: "hexpm", optional: true]}, {:decimal, "~> 1.9 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}], "hexpm", "3844a0a0ed7f42e6590ddd8bd37eb4b1556b112898f67dea3ba068c29aabd6c2"},
8
+
"con_cache": {:hex, :con_cache, "1.1.1", "9f47a68dfef5ac3bbff8ce2c499869dbc5ba889dadde6ac4aff8eb78ddaf6d82", [:mix], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "1def4d1bec296564c75b5bbc60a19f2b5649d81bfa345a2febcc6ae380e8ae15"},
9
+
"cowlib": {:hex, :cowlib, "2.16.0", "54592074ebbbb92ee4746c8a8846e5605052f29309d3a873468d76cdf932076f", [:make, :rebar3], [], "hexpm", "7f478d80d66b747344f0ea7708c187645cfcc08b11aa424632f78e25bf05db51"},
10
+
"credo": {:hex, :credo, "1.7.15", "283da72eeb2fd3ccf7248f4941a0527efb97afa224bcdef30b4b580bc8258e1c", [:mix], [{:bunt, "~> 0.2.1 or ~> 1.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2 or ~> 1.0", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "291e8645ea3fea7481829f1e1eb0881b8395db212821338e577a90bf225c5607"},
11
+
"decimal": {:hex, :decimal, "2.3.0", "3ad6255aa77b4a3c4f818171b12d237500e63525c2fd056699967a3e7ea20f62", [:mix], [], "hexpm", "a4d66355cb29cb47c3cf30e71329e58361cfcb37c34235ef3bf1d7bf3773aeac"},
8
12
"earmark_parser": {:hex, :earmark_parser, "1.4.44", "f20830dd6b5c77afe2b063777ddbbff09f9759396500cdbe7523efd58d7a339c", [:mix], [], "hexpm", "4778ac752b4701a5599215f7030989c989ffdc4f6df457c5f36938cc2d2a2750"},
9
-
"ex_doc": {:hex, :ex_doc, "0.38.2", "504d25eef296b4dec3b8e33e810bc8b5344d565998cd83914ffe1b8503737c02", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "732f2d972e42c116a70802f9898c51b54916e542cc50968ac6980512ec90f42b"},
10
-
"file_system": {:hex, :file_system, "1.1.0", "08d232062284546c6c34426997dd7ef6ec9f8bbd090eb91780283c9016840e8f", [:mix], [], "hexpm", "bfcf81244f416871f2a2e15c1b515287faa5db9c6bcf290222206d120b3d43f6"},
13
+
"ex_cldr": {:hex, :ex_cldr, "2.44.1", "0d220b175874e1ce77a0f7213bdfe700b9be11aefbf35933a0e98837803ebdc5", [:mix], [{:cldr_utils, "~> 2.28", [hex: :cldr_utils, repo: "hexpm", optional: false]}, {:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:gettext, "~> 0.19 or ~> 1.0", [hex: :gettext, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:nimble_parsec, "~> 0.5 or ~> 1.0", [hex: :nimble_parsec, repo: "hexpm", optional: true]}], "hexpm", "3880cd6137ea21c74250cd870d3330c4a9fdec07fabd5e37d1b239547929e29b"},
14
+
"ex_doc": {:hex, :ex_doc, "0.39.3", "519c6bc7e84a2918b737aec7ef48b96aa4698342927d080437f61395d361dcee", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "0590955cf7ad3b625780ee1c1ea627c28a78948c6c0a9b0322bd976a079996e1"},
15
+
"ezstd": {:hex, :ezstd, "1.2.3", "98748f4099e6e2a067f77ace43041ebaa53c13194b08ce22370e4c93079e9e16", [:rebar3], [], "hexpm", "de32e0b41ba36a9ed46db8215da74777d2f141bb75f67bfc05dbb4b7c3386dee"},
16
+
"file_system": {:hex, :file_system, "1.1.1", "31864f4685b0148f25bd3fbef2b1228457c0c89024ad67f7a81a3ffbc0bbad3a", [:mix], [], "hexpm", "7a15ff97dfe526aeefb090a7a9d3d03aa907e100e262a0f8f7746b78f8f87a5d"},
17
+
"finch": {:hex, :finch, "0.20.0", "5330aefb6b010f424dcbbc4615d914e9e3deae40095e73ab0c1bb0968933cadf", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.6.2 or ~> 1.7", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 1.1", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "2658131a74d051aabfcba936093c903b8e89da9a1b63e430bee62045fa9b2ee2"},
11
18
"gun": {:hex, :gun, "2.2.0", "b8f6b7d417e277d4c2b0dc3c07dfdf892447b087f1cc1caff9c0f556b884e33d", [:make, :rebar3], [{:cowlib, ">= 2.15.0 and < 3.0.0", [hex: :cowlib, repo: "hexpm", optional: false]}], "hexpm", "76022700c64287feb4df93a1795cff6741b83fb37415c40c34c38d2a4645261a"},
19
+
"hpax": {:hex, :hpax, "1.0.3", "ed67ef51ad4df91e75cc6a1494f851850c0bd98ebc0be6e81b026e765ee535aa", [:mix], [], "hexpm", "8eab6e1cfa8d5918c2ce4ba43588e894af35dbd8e91e6e55c817bca5847df34a"},
12
20
"jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"},
21
+
"jose": {:hex, :jose, "1.11.12", "06e62b467b61d3726cbc19e9b5489f7549c37993de846dfb3ee8259f9ed208b3", [:mix, :rebar3], [], "hexpm", "31e92b653e9210b696765cdd885437457de1add2a9011d92f8cf63e4641bab7b"},
13
22
"makeup": {:hex, :makeup, "1.2.1", "e90ac1c65589ef354378def3ba19d401e739ee7ee06fb47f94c687016e3713d1", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "d36484867b0bae0fea568d10131197a4c2e47056a6fbe84922bf6ba71c8d17ce"},
14
23
"makeup_elixir": {:hex, :makeup_elixir, "1.0.1", "e928a4f984e795e41e3abd27bfc09f51db16ab8ba1aebdba2b3a575437efafc2", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.2.3 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "7284900d412a3e5cfd97fdaed4f5ed389b8f2b4cb49efc0eb3bd10e2febf9507"},
15
24
"makeup_erlang": {:hex, :makeup_erlang, "1.0.2", "03e1804074b3aa64d5fad7aa64601ed0fb395337b982d9bcf04029d68d51b6a7", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "af33ff7ef368d5893e4a267933e7744e46ce3cf1f61e2dccf53a111ed3aa3727"},
25
+
"mime": {:hex, :mime, "2.0.7", "b8d739037be7cd402aee1ba0306edfdef982687ee7e9859bee6198c1e7e2f128", [:mix], [], "hexpm", "6171188e399ee16023ffc5b76ce445eb6d9672e2e241d2df6050f3c771e80ccd"},
26
+
"mint": {:hex, :mint, "1.7.1", "113fdb2b2f3b59e47c7955971854641c61f378549d73e829e1768de90fc1abf1", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "fceba0a4d0f24301ddee3024ae116df1c3f4bb7a563a731f45fdfeb9d39a231b"},
27
+
"multiformats_ex": {:hex, :multiformats_ex, "0.2.0", "5b0a3faa1a770dc671aa8a89b6323cc20b0ecf67dc93dcd21312151fbea6b4ee", [:mix], [{:varint, "~> 1.4", [hex: :varint, repo: "hexpm", optional: false]}], "hexpm", "aa406d9addb06dc197e0e92212992486af6599158d357680f29f2d11e08d0423"},
28
+
"mutex": {:hex, :mutex, "3.0.2", "528877fd0dbc09fc93ad667e10ea0d35a2126fa85205822f9dca85e87d732245", [:mix], [], "hexpm", "0a8f2ed3618160dca6a1e3520b293dc3c2ae53116265e71b4a732d35d29aa3c6"},
29
+
"nimble_options": {:hex, :nimble_options, "1.1.1", "e3a492d54d85fc3fd7c5baf411d9d2852922f66e69476317787a7b2bb000a61b", [:mix], [], "hexpm", "821b2470ca9442c4b6984882fe9bb0389371b8ddec4d45a9504f00a66f650b44"},
16
30
"nimble_parsec": {:hex, :nimble_parsec, "1.4.2", "8efba0122db06df95bfaa78f791344a89352ba04baedd3849593bfce4d0dc1c6", [:mix], [], "hexpm", "4b21398942dda052b403bbe1da991ccd03a053668d147d53fb8c4e0efe09c973"},
17
-
"typedstruct": {:hex, :typedstruct, "0.5.3", "d68ae424251a41b81a8d0c485328ab48edbd3858f3565bbdac21b43c056fc9b4", [:make, :mix], [], "hexpm", "b53b8186701417c0b2782bf02a2db5524f879b8488f91d1d83b97d84c2943432"},
31
+
"nimble_pool": {:hex, :nimble_pool, "1.1.0", "bf9c29fbdcba3564a8b800d1eeb5a3c58f36e1e11d7b7fb2e084a643f645f06b", [:mix], [], "hexpm", "af2e4e6b34197db81f7aad230c1118eac993acc0dae6bc83bac0126d4ae0813a"},
32
+
"peri": {:hex, :peri, "0.6.2", "3c043bfb6aa18eb1ea41d80981d19294c5e943937b1311e8e958da3581139061", [:mix], [{:ecto, "~> 3.12", [hex: :ecto, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:stream_data, "~> 1.1", [hex: :stream_data, repo: "hexpm", optional: true]}], "hexpm", "5e0d8e0bd9de93d0f8e3ad6b9a5bd143f7349c025196ef4a3591af93ce6ecad9"},
33
+
"plug": {:hex, :plug, "1.19.1", "09bac17ae7a001a68ae393658aa23c7e38782be5c5c00c80be82901262c394c0", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "560a0017a8f6d5d30146916862aaf9300b7280063651dd7e532b8be168511e62"},
34
+
"plug_crypto": {:hex, :plug_crypto, "2.1.1", "19bda8184399cb24afa10be734f84a16ea0a2bc65054e23a62bb10f06bc89491", [:mix], [], "hexpm", "6470bce6ffe41c8bd497612ffde1a7e4af67f36a15eea5f921af71cf3e11247c"},
35
+
"recase": {:hex, :recase, "0.9.1", "82d2e2e2d4f9e92da1ce5db338ede2e4f15a50ac1141fc082b80050b9f49d96e", [:mix], [], "hexpm", "19ba03ceb811750e6bec4a015a9f9e45d16a8b9e09187f6d72c3798f454710f3"},
36
+
"req": {:hex, :req, "0.5.17", "0096ddd5b0ed6f576a03dde4b158a0c727215b15d2795e59e0916c6971066ede", [:mix], [{:brotli, "~> 0.3.1", [hex: :brotli, repo: "hexpm", optional: true]}, {:ezstd, "~> 1.0", [hex: :ezstd, repo: "hexpm", optional: true]}, {:finch, "~> 0.17", [hex: :finch, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mime, "~> 2.0.6 or ~> 2.1", [hex: :mime, repo: "hexpm", optional: false]}, {:nimble_csv, "~> 1.0", [hex: :nimble_csv, repo: "hexpm", optional: true]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "0b8bc6ffdfebbc07968e59d3ff96d52f2202d0536f10fef4dc11dc02a2a43e39"},
37
+
"telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
38
+
"typedstruct": {:hex, :typedstruct, "0.5.4", "d1d33d58460a74f413e9c26d55e66fd633abd8ac0fb12639add9a11a60a0462a", [:make, :mix], [], "hexpm", "ffaef36d5dbaebdbf4ed07f7fb2ebd1037b2c1f757db6fb8e7bcbbfabbe608d8"},
18
39
"varint": {:hex, :varint, "1.5.1", "17160c70d0428c3f8a7585e182468cac10bbf165c2360cf2328aaa39d3fb1795", [:mix], [], "hexpm", "24f3deb61e91cb988056de79d06f01161dd01be5e0acae61d8d936a552f1be73"},
19
40
}
priv/jetstream/zstd_dictionary
priv/jetstream/zstd_dictionary
This is a binary file and will not be displayed.