atproto libraries implementation in ocaml

feat(examples): add AT Protocol example tools and GUIDE

Add four example tools demonstrating AT Protocol library usage:

- bsky_bot: Bluesky bot for posting, timeline, profiles, following
- feed_generator: Firehose client with filtering and feed skeleton
(evolved from firehose_demo with keyword filtering)
- identity_tool: Handle/DID resolution and identity verification
- repo_inspector: Repository exploration with auto PDS resolution

Key improvements:
- repo_inspector now auto-resolves PDS from DID document
- Added --pds option to skip auto-resolution
- Fixed URL encoding issue (? was incorrectly encoded as %3F)

Add GUIDE.md showing complete workflow: firehose -> identity -> repo

Remove deprecated firehose_demo in favor of feed_generator.

Changed files
+1771 -495
.beads
examples
+4
.beads/issues.jsonl
··· 32 32 {"id":"atproto-61","title":"Set up interoperability test suite","description":"Set up and run the AT Protocol interoperability tests from bluesky-social/atproto-interop-tests.","design":"## Test Structure\n\n```\ntest/\n├── interop/\n│ ├── syntax_test.ml # Handle, DID, NSID, TID, etc.\n│ ├── crypto_test.ml # Signatures, did:key\n│ ├── data_model_test.ml # DAG-CBOR, CID\n│ ├── mst_test.ml # Key heights, tree structure\n│ ├── lexicon_test.ml # Schema and record validation\n│ └── firehose_test.ml # Commit proofs\n├── fixtures/ # Cloned from atproto-interop-tests\n└── dune\n```\n\n## Test Approach\n\n1. Clone test vectors from GitHub\n2. Parse JSON fixtures using jsont\n3. Parse text fixtures line by line\n4. Run each test case\n5. Compare output to expected values\n\n## Example Test\n\n```ocaml\nlet load_json_fixtures path =\n let json = Jsont.of_file path in\n Jsont.decode (Jsont.list fixture_jsont) json\n\nlet%test \"handle_syntax_valid\" =\n let fixtures = load_lines \"fixtures/syntax/handle_syntax_valid.txt\" in\n List.for_all (fun line -\u003e\n match Handle.of_string line with\n | Ok _ -\u003e true\n | Error _ -\u003e false\n ) fixtures\n\nlet%test \"handle_syntax_invalid\" =\n let fixtures = load_lines \"fixtures/syntax/handle_syntax_invalid.txt\" in\n List.for_all (fun line -\u003e\n match Handle.of_string line with\n | Ok _ -\u003e false\n | Error _ -\u003e true\n ) fixtures\n\nlet%test \"crypto_signature_fixtures\" =\n let fixtures = load_json_fixtures \"fixtures/crypto/signature-fixtures.json\" in\n List.for_all (fun fixture -\u003e\n let message = Base64.decode fixture.message_base64 in\n let signature = Base64.decode fixture.signature_base64 in\n let key = Did_key.of_string fixture.public_key_did in\n let result = Crypto.verify key message signature in\n result = fixture.valid_signature\n ) fixtures\n```\n\n## Dependencies\n- alcotest or ounit2\n- jsont","acceptance_criteria":"- All syntax interop tests pass\n- All crypto interop tests pass\n- All data-model interop tests pass\n- All MST interop tests pass\n- All lexicon interop tests pass\n- All firehose interop tests pass","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-28T00:12:40.553908313+01:00","updated_at":"2025-12-28T13:25:34.614867702+01:00","closed_at":"2025-12-28T13:25:34.614867702+01:00","labels":["conformance","testing"],"dependencies":[{"issue_id":"atproto-61","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T00:12:56.180809368+01:00","created_by":"daemon"}]} 33 33 {"id":"atproto-62","title":"Set up monorepo package structure","description":"Set up the monorepo structure for multiple opam packages within a single repository.","design":"## Repository Structure\n\n```\natproto/\n├── dune-project # Root with all packages\n├── packages/\n│ ├── atproto-syntax/\n│ │ ├── lib/\n│ │ │ ├── dune\n│ │ │ └── *.ml\n│ │ ├── test/\n│ │ │ ├── dune\n│ │ │ └── *_test.ml\n│ │ └── atproto-syntax.opam\n│ ├── atproto-crypto/\n│ ├── atproto-multibase/\n│ ├── atproto-ipld/\n│ ├── atproto-mst/\n│ ├── atproto-repo/\n│ ├── atproto-identity/\n│ ├── atproto-xrpc/\n│ ├── atproto-sync/\n│ ├── atproto-lexicon/\n│ ├── atproto-lexicon-gen/\n│ ├── atproto-api/\n│ └── atproto-effects/\n├── examples/\n│ ├── simple_client/\n│ └── firehose_consumer/\n└── interop-tests/\n```\n\n## dune-project\n\n```lisp\n(lang dune 3.20)\n(name atproto)\n(generate_opam_files true)\n\n(package\n (name atproto-syntax)\n (synopsis \"AT Protocol identifier syntax parsing\")\n (depends\n (ocaml (\u003e= 5.4))\n re\n ptime))\n\n(package\n (name atproto-crypto)\n ...)\n```\n\n## CI (.github/workflows/ci.yml)\n\n- OCaml 5.4 matrix\n- Build all packages\n- Run all tests\n- Run interop tests","acceptance_criteria":"- Multi-package dune-project structure\n- Separate opam files per package\n- CI pipeline for building and testing\n- Documentation generation setup","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-28T00:12:50.547102438+01:00","updated_at":"2025-12-28T11:57:18.856810633+01:00","closed_at":"2025-12-28T11:57:18.856810633+01:00","labels":["infrastructure","setup"],"dependencies":[{"issue_id":"atproto-62","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T00:12:57.015938611+01:00","created_by":"daemon"}]} 34 34 {"id":"atproto-cir","title":"Implement DAG-JSON codec","description":"Create lib/ipld/dag_json.ml, encode Links as {\"/\": \"\u003ccid-string\u003e\"}, encode Bytes as {\"/\": {\"bytes\": \"\u003cbase64\u003e\"}}","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-12-28T15:47:30.578398468+01:00","updated_at":"2025-12-28T16:06:13.475583417+01:00","closed_at":"2025-12-28T16:06:13.475583417+01:00","labels":["compliance","ipld"]} 35 + {"id":"atproto-ex-bot","title":"Implement Bluesky Bot Example (bsky_bot)","description":"Bot that can login, post with rich text, reply to mentions, follow back. Uses atproto-api Agent, Richtext, atproto-xrpc Client. Eio for I/O, Climate for CLI.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-29T09:07:59.407148786+01:00","updated_at":"2025-12-29T09:16:27.789416628+01:00","closed_at":"2025-12-29T09:16:27.789416628+01:00","labels":["api","bot","cli","example"]} 36 + {"id":"atproto-ex-feed","title":"Implement Feed Generator Example (feed_generator)","description":"Custom feed algorithm with XRPC server serving app.bsky.feed.getFeedSkeleton. Subscribes to firehose, filters posts by keyword/language. Uses atproto-xrpc Server, atproto-sync Firehose.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-29T09:08:01.619973249+01:00","updated_at":"2025-12-29T09:18:53.434611454+01:00","closed_at":"2025-12-29T09:18:53.434611454+01:00","labels":["example","feed","firehose","server"]} 37 + {"id":"atproto-ex-identity","title":"Implement Handle/DID Lookup Tool (identity_tool)","description":"CLI tool using Climate to resolve handles to DIDs, DIDs to documents, verify bidirectional identity links. Uses atproto-identity, atproto-syntax. Eio for I/O.","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-29T09:07:57.316042741+01:00","updated_at":"2025-12-29T09:12:20.634271257+01:00","closed_at":"2025-12-29T09:12:20.634271257+01:00","labels":["cli","example","identity"]} 38 + {"id":"atproto-ex-repo","title":"Implement Repository Inspector Example (repo_inspector)","description":"Tool to explore AT Protocol repositories: download CAR, parse MST structure, show records with CIDs, verify commit signatures. Uses atproto-repo, atproto-mst, atproto-ipld, atproto-crypto.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-29T09:08:03.622391028+01:00","updated_at":"2025-12-29T09:21:32.223029796+01:00","closed_at":"2025-12-29T09:21:32.223029796+01:00","labels":["cli","educational","example","repo"]} 35 39 {"id":"atproto-fw8","title":"Add filter option to firehose demo","description":"Add --filter option to firehose demo to filter events by type (posts, likes, follows, reposts, profiles, blocks, lists, etc.). This makes it easier to monitor specific activity types.","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-12-28T21:08:20.29215305+01:00","updated_at":"2025-12-28T21:13:31.957432542+01:00","closed_at":"2025-12-28T21:13:31.957432542+01:00","labels":["demo","enhancement","firehose"]} 36 40 {"id":"atproto-h09","title":"Add package documentation","description":"Add documentation for each of the 11 AT Protocol packages including:\n- Module-level documentation with examples\n- README.md for the root project\n- CONTRIBUTING.md guide\n- API documentation generation with odoc","status":"closed","priority":2,"issue_type":"task","assignee":"claude","created_at":"2025-12-28T13:34:10.559554696+01:00","updated_at":"2025-12-28T13:50:20.509417248+01:00","closed_at":"2025-12-28T13:50:20.509417248+01:00","labels":["documentation"],"dependencies":[{"issue_id":"atproto-h09","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T13:34:16.081103184+01:00","created_by":"daemon"}]} 37 41 {"id":"atproto-kc7","title":"Verify/enforce CBOR strictness","description":"Check that CBOR library produces shortest-form encoding, add explicit rejection of indefinite-length items during decode","status":"closed","priority":2,"issue_type":"feature","created_at":"2025-12-28T15:47:28.665226526+01:00","updated_at":"2025-12-28T16:02:55.549763384+01:00","closed_at":"2025-12-28T16:02:55.549763384+01:00","labels":["compliance","ipld"]}
+489
examples/GUIDE.md
··· 1 + # AT Protocol Tools Guide 2 + 3 + This guide demonstrates how to use the example tools together to explore the AT Protocol ecosystem. We'll walk through a complete workflow: finding interesting posts in the firehose, investigating user identities, and inspecting their repositories. 4 + 5 + ## Prerequisites 6 + 7 + Build all examples: 8 + 9 + ```bash 10 + dune build examples/ 11 + ``` 12 + 13 + Set up your credentials for `bsky_bot` (required for authenticated operations): 14 + 15 + ```bash 16 + export BSKY_USER="yourhandle.bsky.social" 17 + export BSKY_PASS="xxxx-xxxx-xxxx-xxxx" # App password from Settings > App Passwords 18 + ``` 19 + 20 + ## Overview of Tools 21 + 22 + | Tool | Purpose | 23 + |------|---------| 24 + | `feed_generator` | Subscribe to the firehose and filter events | 25 + | `identity_tool` | Resolve handles/DIDs and verify identities | 26 + | `repo_inspector` | Download and explore user repositories | 27 + | `bsky_bot` | Interact with Bluesky (post, timeline, profiles) | 28 + 29 + --- 30 + 31 + ## Step 1: Find Interesting Posts with feed_generator 32 + 33 + Start by subscribing to the firehose and filtering for posts containing a keyword of interest. Let's look for posts about "ocaml": 34 + 35 + ```bash 36 + `dune exec examples/feed_generator/feed_generator.exe -- \ 37 + --filter posts \ 38 + --keyword "ocaml" \ 39 + --limit 5 \ 40 + --content` 41 + ``` 42 + 43 + Example output: 44 + 45 + ``` 46 + Feed Generator 47 + ============== 48 + 49 + Type filter: posts 50 + Keyword: "ocaml" 51 + 52 + Connecting to wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos... 53 + Connected! 54 + 55 + #commit seq=12345678 repo=did:plc:abc123xyz456 56 + +app.bsky.feed.post/3abc123 "Just discovered OCaml! The type system is amazing..." 57 + #commit seq=12345680 repo=did:plc:def789uvw012 58 + +app.bsky.feed.post/3def456 "Working on my first OCaml project today..." 59 + 60 + --- Stats: 1523 events (5 matched) | seq=12345700 | 8.2s | 185.7 evt/s --- 61 + ``` 62 + 63 + **Key information extracted:** 64 + - `did:plc:abc123xyz456` - The DID of a user who posted about OCaml 65 + - `app.bsky.feed.post/3abc123` - The record key of their post 66 + 67 + ### Generating a Feed Skeleton 68 + 69 + If you want to build a custom feed, use the `--skeleton` flag to output a feed skeleton: 70 + 71 + ```bash 72 + dune exec examples/feed_generator/feed_generator.exe -- \ 73 + --filter posts \ 74 + --keyword "rust" \ 75 + --limit 20 \ 76 + --skeleton 77 + ``` 78 + 79 + This outputs a JSON structure you can serve via `app.bsky.feed.getFeedSkeleton`: 80 + 81 + ```json 82 + { 83 + "feed": [ 84 + {"post": "at://did:plc:xxx/app.bsky.feed.post/abc123"}, 85 + {"post": "at://did:plc:yyy/app.bsky.feed.post/def456"} 86 + ] 87 + } 88 + ``` 89 + 90 + ### Monitoring Multiple Event Types 91 + 92 + Watch for posts, likes, and follows simultaneously: 93 + 94 + ```bash 95 + dune exec examples/feed_generator/feed_generator.exe -- \ 96 + --filter posts,likes,follows \ 97 + --limit 50 98 + ``` 99 + 100 + --- 101 + 102 + ## Step 2: Verify User Identity with identity_tool 103 + 104 + Now that we have a DID from the firehose, let's verify the identity and get more details about the user. 105 + 106 + ### Verify Identity (Bidirectional) 107 + 108 + The default mode verifies that the DID and handle point to each other: 109 + 110 + ```bash 111 + dune exec examples/identity_tool/identity_tool.exe -- did:plc:abc123xyz456 112 + ``` 113 + 114 + Example output: 115 + 116 + ``` 117 + PASSED 118 + DID: did:plc:abc123xyz456 119 + Handle: alice.bsky.social 120 + Key: zQ3shXjHeiBuRCKmM... 121 + PDS: https://morel.us-east.host.bsky.network 122 + ``` 123 + 124 + This tells us: 125 + - The identity verification passed (handle and DID are correctly linked) 126 + - The user's handle is `alice.bsky.social` 127 + - Their signing key (for verifying commits) 128 + - Their PDS (Personal Data Server) URL 129 + 130 + ### Resolve Handle to DID 131 + 132 + If you have a handle and need the DID: 133 + 134 + ```bash 135 + dune exec examples/identity_tool/identity_tool.exe -- -H alice.bsky.social 136 + ``` 137 + 138 + Output: 139 + 140 + ``` 141 + Handle: alice.bsky.social 142 + DID: did:plc:abc123xyz456 143 + ``` 144 + 145 + ### Resolve DID to Full Document 146 + 147 + Get the complete DID document with all services and keys: 148 + 149 + ```bash 150 + dune exec examples/identity_tool/identity_tool.exe -- -d did:plc:abc123xyz456 151 + ``` 152 + 153 + Output: 154 + 155 + ``` 156 + DID: did:plc:abc123xyz456 157 + Handle: at://alice.bsky.social 158 + Key: zQ3shXjHeiBuRCKmM... (Multikey) 159 + Service: AtprotoPersonalDataServer -> https://morel.us-east.host.bsky.network 160 + ``` 161 + 162 + --- 163 + 164 + ## Step 3: Inspect User Repository with repo_inspector 165 + 166 + With the verified DID, we can download and inspect the user's entire repository. The tool automatically resolves the user's PDS from their DID document. 167 + 168 + ### Repository Summary 169 + 170 + Get an overview of the repository: 171 + 172 + ```bash 173 + dune exec examples/repo_inspector/repo_inspector.exe -- did:plc:abc123xyz456 174 + ``` 175 + 176 + Example output: 177 + 178 + ``` 179 + Repository Inspector 180 + ==================== 181 + 182 + Resolving PDS for did:plc:abc123xyz456... 183 + PDS: https://morel.us-east.host.bsky.network 184 + 185 + Fetching https://morel.us-east.host.bsky.network/xrpc/com.atproto.sync.getRepo?did=did:plc:abc123xyz456... 186 + 187 + CAR Header 188 + ---------- 189 + Version: 1 190 + Roots: 1 191 + - bafyreib... 192 + Blocks: 1234 193 + 194 + Commit 195 + ------ 196 + DID: did:plc:abc123xyz456 197 + Version: 3 198 + Rev: 3la7j2xyz... 199 + Data: bafyreic... 200 + 201 + Collections 202 + ----------- 203 + app.bsky.feed.post: 456 204 + app.bsky.feed.like: 789 205 + app.bsky.graph.follow: 123 206 + app.bsky.actor.profile: 1 207 + 208 + Total records: 1369 209 + ``` 210 + 211 + ### List Collections 212 + 213 + See all collections with record counts: 214 + 215 + ```bash 216 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 217 + did:plc:abc123xyz456 --collections 218 + ``` 219 + 220 + ### Browse Records 221 + 222 + View individual records: 223 + 224 + ```bash 225 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 226 + did:plc:abc123xyz456 --records --limit 10 227 + ``` 228 + 229 + Output: 230 + 231 + ``` 232 + Records 233 + ------- 234 + app.bsky.feed.post/3abc123 235 + CID: bafyreig... 236 + Text: Just discovered OCaml! The type system is amazing... 237 + 238 + app.bsky.feed.post/3def456 239 + CID: bafyreih... 240 + Text: Working on my second OCaml project... 241 + 242 + ... and 454 more 243 + ``` 244 + 245 + ### Filter by Collection 246 + 247 + View only posts: 248 + 249 + ```bash 250 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 251 + did:plc:abc123xyz456 --records --collection app.bsky.feed.post --limit 20 252 + ``` 253 + 254 + View only follows: 255 + 256 + ```bash 257 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 258 + did:plc:abc123xyz456 --records --collection app.bsky.graph.follow --limit 20 259 + ``` 260 + 261 + ### Verify Commit Signature 262 + 263 + Check the repository commit signature: 264 + 265 + ```bash 266 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 267 + did:plc:abc123xyz456 --verify 268 + ``` 269 + 270 + ### Skip PDS Resolution 271 + 272 + If you already know the PDS URL (from identity_tool or a previous run), you can skip the resolution step: 273 + 274 + ```bash 275 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 276 + did:plc:abc123xyz456 --pds https://morel.us-east.host.bsky.network 277 + ``` 278 + 279 + --- 280 + 281 + ## Step 4: Get User Profile with bsky_bot 282 + 283 + For a more user-friendly view, use `bsky_bot` to get the profile (requires authentication): 284 + 285 + ```bash 286 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 287 + --user "$BSKY_USER" \ 288 + --password "$BSKY_PASS" \ 289 + --profile alice.bsky.social 290 + ``` 291 + 292 + Output: 293 + 294 + ``` 295 + Logged in as yourhandle.bsky.social 296 + Handle: @alice.bsky.social 297 + DID: did:plc:abc123xyz456 298 + Name: Alice 299 + Bio: OCaml enthusiast and functional programming advocate 300 + Followers: 1234 301 + Following: 567 302 + Posts: 456 303 + ``` 304 + 305 + ### View Your Timeline 306 + 307 + See recent posts from people you follow: 308 + 309 + ```bash 310 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 311 + --user "$BSKY_USER" \ 312 + --password "$BSKY_PASS" \ 313 + --timeline --limit 10 314 + ``` 315 + 316 + ### Follow a User 317 + 318 + Follow someone using their DID (obtained from identity_tool or feed_generator): 319 + 320 + ```bash 321 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 322 + --user "$BSKY_USER" \ 323 + --password "$BSKY_PASS" \ 324 + --follow did:plc:abc123xyz456 325 + ``` 326 + 327 + --- 328 + 329 + ## Complete Workflow Example 330 + 331 + Here's a complete example workflow to find OCaml enthusiasts and explore their content: 332 + 333 + ```bash 334 + #!/bin/bash 335 + 336 + # 1. Find users posting about OCaml 337 + echo "=== Finding OCaml posts ===" 338 + dune exec examples/feed_generator/feed_generator.exe -- \ 339 + --filter posts --keyword "ocaml" --limit 3 --json 2>/dev/null | \ 340 + head -3 341 + 342 + # Let's say we found: did:plc:z72i7hdynmk6r22z27h6tvur 343 + 344 + DID="did:plc:z72i7hdynmk6r22z27h6tvur" 345 + 346 + # 2. Verify the identity 347 + echo -e "\n=== Verifying identity ===" 348 + dune exec examples/identity_tool/identity_tool.exe -- "$DID" 349 + 350 + # 3. Get their handle from the verification and look up their profile 351 + HANDLE="jay.bsky.social" # Extracted from step 2 352 + 353 + echo -e "\n=== Getting profile ===" 354 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 355 + --user "$BSKY_USER" --password "$BSKY_PASS" \ 356 + --profile "$HANDLE" 357 + 358 + # 4. Inspect their repository 359 + echo -e "\n=== Repository overview ===" 360 + dune exec examples/repo_inspector/repo_inspector.exe -- "$DID" 361 + 362 + # 5. See their recent posts 363 + echo -e "\n=== Recent posts ===" 364 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 365 + "$DID" --records --collection app.bsky.feed.post --limit 5 366 + ``` 367 + 368 + --- 369 + 370 + ## Advanced Usage 371 + 372 + ### JSON Output for Scripting 373 + 374 + The feed_generator supports JSON output for integration with other tools: 375 + 376 + ```bash 377 + dune exec examples/feed_generator/feed_generator.exe -- \ 378 + --filter posts --keyword "atproto" --limit 5 --json --content 379 + ``` 380 + 381 + You can pipe this to `jq` for further processing: 382 + 383 + ```bash 384 + dune exec examples/feed_generator/feed_generator.exe -- \ 385 + --filter posts --limit 10 --json 2>/dev/null | \ 386 + jq -r '.repo' | sort | uniq 387 + ``` 388 + 389 + ### Resume from Cursor 390 + 391 + If you need to resume from a specific point in the firehose: 392 + 393 + ```bash 394 + dune exec examples/feed_generator/feed_generator.exe -- \ 395 + --cursor 1234567890 --limit 100 396 + ``` 397 + 398 + ### Monitoring Identity Changes 399 + 400 + Watch for handle changes and identity updates: 401 + 402 + ```bash 403 + dune exec examples/feed_generator/feed_generator.exe -- \ 404 + --filter identities,handles --limit 20 405 + ``` 406 + 407 + ### Watching Account Events 408 + 409 + Monitor account activations and deactivations: 410 + 411 + ```bash 412 + dune exec examples/feed_generator/feed_generator.exe -- \ 413 + --filter accounts --limit 10 414 + ``` 415 + 416 + --- 417 + 418 + ## Architecture Overview 419 + 420 + ``` 421 + +-------------------+ 422 + | Bluesky Relay | 423 + | bsky.network | 424 + +-------------------+ 425 + | 426 + | WebSocket (firehose) 427 + v 428 + +-------------------+ 429 + | feed_generator | 430 + | (filter events) | 431 + +-------------------+ 432 + | 433 + | DID extracted 434 + v 435 + +-------------------+ +-------------------+ +-------------------+ 436 + | identity_tool |<---->| PLC Directory | | User's PDS | 437 + | (resolve/verify) | | DID Documents | | | 438 + +-------------------+ +-------------------+ +-------------------+ 439 + | ^ 440 + | DID verified | 441 + v | 442 + +-------------------+ | 443 + | repo_inspector |------------------------------------------+ 444 + | (download & parse)| com.atproto.sync.getRepo 445 + +-------------------+ 446 + 447 + +-------------------+ 448 + | bsky_bot |-------> Authenticated XRPC calls 449 + | (social actions) | (post, timeline, profile, follow) 450 + +-------------------+ 451 + ``` 452 + 453 + ## Troubleshooting 454 + 455 + ### Connection Errors 456 + 457 + If the firehose connection fails: 458 + - Check your internet connection 459 + - The relay at `bsky.network` may be temporarily unavailable 460 + - Try again after a few seconds 461 + 462 + ### Identity Resolution Failures 463 + 464 + If identity verification fails: 465 + - The handle may have changed recently (DNS propagation delay) 466 + - The DID document may be temporarily unavailable 467 + - Try resolving the handle and DID separately to diagnose 468 + 469 + ### Large Repositories 470 + 471 + For users with many records: 472 + - Use `--limit` to restrict the number of records shown 473 + - Use `--collection` to filter to specific record types 474 + - Repository download may take several seconds for active users 475 + 476 + --- 477 + 478 + ## Next Steps 479 + 480 + - Build a custom feed generator using `feed_generator` as a template 481 + - Create a bot that responds to mentions using `bsky_bot` 482 + - Build analytics tools using `repo_inspector` to analyze user behavior 483 + - Implement identity verification in your applications using `identity_tool` 484 + 485 + See the individual README files in each example directory for more details: 486 + - [bsky_bot/README.md](bsky_bot/README.md) 487 + - [feed_generator/README.md](feed_generator/README.md) 488 + - [identity_tool/README.md](identity_tool/README.md) 489 + - [repo_inspector/README.md](repo_inspector/README.md)
+91
examples/bsky_bot/README.md
··· 1 + # bsky_bot 2 + 3 + Bluesky bot for posting, viewing timelines, fetching profiles, and following users via AT Protocol. 4 + 5 + ## Usage 6 + 7 + ``` 8 + bsky_bot --user <HANDLE> --password <APP_PASSWORD> [COMMAND] 9 + ``` 10 + 11 + Authentication is required for all operations. Use an App Password from your Bluesky account settings. 12 + 13 + ## Options 14 + 15 + | Option | Description | 16 + |--------|-------------| 17 + | `--pds <URL>` | PDS URL (default: https://bsky.social) | 18 + | `-u, --user <STRING>` | Handle or email for login | 19 + | `-p, --password <STRING>` | App password | 20 + | `--post <TEXT>` | Create a post with the given text | 21 + | `-t, --timeline` | Show timeline | 22 + | `--limit <INT>` | Number of timeline posts (default: 10) | 23 + | `--profile <ACTOR>` | Get profile for handle or DID | 24 + | `--follow <DID>` | Follow a user by DID | 25 + | `-h, --help` | Show help message | 26 + 27 + ## Examples 28 + 29 + Post a message: 30 + 31 + ```bash 32 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 33 + --user myhandle.bsky.social \ 34 + --password xxxx-xxxx-xxxx-xxxx \ 35 + --post "Hello from OCaml!" 36 + ``` 37 + 38 + View your timeline: 39 + 40 + ```bash 41 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 42 + --user myhandle.bsky.social \ 43 + --password xxxx-xxxx-xxxx-xxxx \ 44 + --timeline --limit 5 45 + ``` 46 + 47 + Get a user's profile: 48 + 49 + ```bash 50 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 51 + --user myhandle.bsky.social \ 52 + --password xxxx-xxxx-xxxx-xxxx \ 53 + --profile jay.bsky.social 54 + ``` 55 + 56 + Follow a user: 57 + 58 + ```bash 59 + dune exec examples/bsky_bot/bsky_bot.exe -- \ 60 + --user myhandle.bsky.social \ 61 + --password xxxx-xxxx-xxxx-xxxx \ 62 + --follow did:plc:z72i7hdynmk6r22z27h6tvur 63 + ``` 64 + 65 + ## Output 66 + 67 + Timeline output: 68 + 69 + ``` 70 + Logged in as myhandle.bsky.social 71 + @jay.bsky.social: Just shipped a new feature... 72 + @alice.bsky.social: Check out this thread about... 73 + ``` 74 + 75 + Profile output: 76 + 77 + ``` 78 + Handle: @jay.bsky.social 79 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 80 + Name: Jay 81 + Bio: Building things 82 + Followers: 12345 83 + Following: 678 84 + Posts: 901 85 + ``` 86 + 87 + ## Notes 88 + 89 + - Posts automatically detect and link mentions (@handle.domain) and URLs 90 + - The `--post` option supports text up to 300 graphemes (Bluesky limit) 91 + - Use App Passwords from Settings > App Passwords, not your main password
+188
examples/bsky_bot/bsky_bot.ml
··· 1 + (** Bluesky Bot - Post, reply, follow operations via AT Protocol. Usage: 2 + bsky_bot [--post|--timeline|--profile|--follow] [options] *) 3 + 4 + module Agent = Atproto_api.Agent 5 + module Richtext = Atproto_api.Richtext 6 + module Client = Atproto_xrpc.Client 7 + 8 + (** {1 HTTP Client with cohttp-eio} *) 9 + 10 + let http_request ~sw ~client (req : Client.request) : Client.response = 11 + let headers = Cohttp.Header.of_list req.headers in 12 + let body = 13 + match req.body with 14 + | Some b -> Cohttp_eio.Body.of_string b 15 + | None -> Cohttp_eio.Body.of_string "" 16 + in 17 + let meth = match req.meth with `GET -> `GET | `POST -> `POST in 18 + try 19 + let resp, resp_body = 20 + Cohttp_eio.Client.call ~sw client meth req.uri ~headers ~body 21 + in 22 + let status = Cohttp.Response.status resp |> Cohttp.Code.code_of_status in 23 + let headers = Cohttp.Response.headers resp |> Cohttp.Header.to_list in 24 + let body = 25 + Eio.Buf_read.(of_flow ~max_size:(10 * 1024 * 1024) resp_body |> take_all) 26 + in 27 + { Client.status; headers; body } 28 + with e -> { Client.status = 0; headers = []; body = Printexc.to_string e } 29 + 30 + (** {1 Effect Handler} *) 31 + 32 + let run_with_eio ~sw ~client f = 33 + Effect.Deep.try_with f () 34 + { 35 + effc = 36 + (fun (type a) (eff : a Effect.t) -> 37 + match eff with 38 + | Client.Http_request req -> 39 + Some 40 + (fun (k : (a, _) Effect.Deep.continuation) -> 41 + Effect.Deep.continue k (http_request ~sw ~client req)) 42 + | _ -> None); 43 + } 44 + 45 + (** {1 Commands} *) 46 + 47 + let login ~sw ~client ~pds ~identifier ~password = 48 + run_with_eio ~sw ~client (fun () -> 49 + let agent = Agent.create_from_url ~url:pds in 50 + match Agent.login agent ~identifier ~password with 51 + | Error e -> 52 + Printf.printf "Login failed: %s\n" (Agent.error_to_string e); 53 + None 54 + | Ok agent -> 55 + Printf.printf "Logged in as %s\n" 56 + (Option.value ~default:"?" (Agent.handle agent)); 57 + Some agent) 58 + 59 + let post ~sw ~client agent text = 60 + run_with_eio ~sw ~client (fun () -> 61 + let rt = Richtext.detect_facets text in 62 + match Agent.create_post_richtext agent ~richtext:rt () with 63 + | Error e -> 64 + Printf.printf "Post failed: %s\n" (Agent.error_to_string e); 65 + 1 66 + | Ok r -> 67 + Printf.printf "Posted: %s\n" r.uri; 68 + 0) 69 + 70 + let timeline ~sw ~client agent limit = 71 + run_with_eio ~sw ~client (fun () -> 72 + match Agent.get_timeline agent ~limit () with 73 + | Error e -> 74 + Printf.printf "Timeline failed: %s\n" (Agent.error_to_string e); 75 + 1 76 + | Ok feed -> 77 + List.iter 78 + (fun (p : Agent.feed_item) -> 79 + Printf.printf "@%s: %s\n" p.author_handle 80 + (String.sub p.text 0 (min 80 (String.length p.text)))) 81 + feed.items; 82 + 0) 83 + 84 + let profile ~sw ~client agent actor = 85 + run_with_eio ~sw ~client (fun () -> 86 + match Agent.get_profile agent ~actor with 87 + | Error e -> 88 + Printf.printf "Profile failed: %s\n" (Agent.error_to_string e); 89 + 1 90 + | Ok p -> 91 + Printf.printf "Handle: @%s\nDID: %s\n" p.handle p.did; 92 + Option.iter (Printf.printf "Name: %s\n") p.display_name; 93 + Option.iter (Printf.printf "Bio: %s\n") p.description; 94 + Printf.printf "Followers: %d\nFollowing: %d\nPosts: %d\n" 95 + p.followers_count p.follows_count p.posts_count; 96 + 0) 97 + 98 + let follow ~sw ~client agent did = 99 + run_with_eio ~sw ~client (fun () -> 100 + match Agent.follow agent ~did with 101 + | Error e -> 102 + Printf.printf "Follow failed: %s\n" (Agent.error_to_string e); 103 + 1 104 + | Ok r -> 105 + Printf.printf "Followed: %s\n" r.uri; 106 + 0) 107 + 108 + (** {1 CLI} *) 109 + 110 + type cmd = 111 + | Post of string 112 + | Timeline of int 113 + | Profile of string 114 + | Follow of string 115 + | None 116 + 117 + let cli = 118 + let open Climate.Arg_parser in 119 + let+ pds = named_opt [ "pds" ] string ~doc:"PDS URL (default: bsky.social)" 120 + and+ identifier = named_opt [ "u"; "user" ] string ~doc:"Handle or email" 121 + and+ password = named_opt [ "p"; "password" ] string ~doc:"App password" 122 + and+ post_text = named_opt [ "post" ] string ~doc:"Post text" 123 + and+ do_timeline = flag [ "timeline"; "t" ] ~doc:"Show timeline" 124 + and+ timeline_limit = 125 + named_opt [ "limit" ] int ~doc:"Timeline limit (default: 10)" 126 + and+ profile_actor = named_opt [ "profile" ] string ~doc:"Get profile" 127 + and+ follow_did = named_opt [ "follow" ] string ~doc:"Follow DID" in 128 + let pds = Option.value pds ~default:"https://bsky.social" in 129 + let limit = Option.value timeline_limit ~default:10 in 130 + let cmd = 131 + match (post_text, do_timeline, profile_actor, follow_did) with 132 + | Some t, _, _, _ -> Post t 133 + | _, true, _, _ -> Timeline limit 134 + | _, _, Some a, _ -> Profile a 135 + | _, _, _, Some d -> Follow d 136 + | _ -> None 137 + in 138 + (pds, identifier, password, cmd) 139 + 140 + let () = 141 + Mirage_crypto_rng_unix.use_default (); 142 + Eio_main.run @@ fun env -> 143 + Eio.Switch.run @@ fun sw -> 144 + let https_config = 145 + match 146 + Tls.Config.client ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) () 147 + with 148 + | Ok c -> c 149 + | Error (`Msg m) -> failwith m 150 + in 151 + let https uri socket = 152 + let tls_host = 153 + match Uri.host uri with 154 + | Some h -> ( 155 + match Domain_name.of_string h with 156 + | Error _ -> Option.None 157 + | Ok dn -> Domain_name.host dn |> Result.to_option) 158 + | Option.None -> Option.None 159 + in 160 + Tls_eio.client_of_flow https_config ?host:tls_host socket 161 + in 162 + let client = Cohttp_eio.Client.make ~https:(Some https) env#net in 163 + let pds, identifier, password, cmd = 164 + Climate.Command.run ~program_name:(Climate.Program_name.Literal "bsky_bot") 165 + (Climate.Command.singleton ~doc:"Bluesky bot - post, timeline, follow" cli) 166 + in 167 + match cmd with 168 + | None -> 169 + Printf.printf 170 + "Usage: bsky_bot --user USER --password PASS [--post \ 171 + TEXT|--timeline|--profile ACTOR|--follow DID]\n"; 172 + exit 0 173 + | _ -> ( 174 + match (identifier, password) with 175 + | Some id, Some pw -> ( 176 + match login ~sw ~client ~pds ~identifier:id ~password:pw with 177 + | None -> exit 1 178 + | Some agent -> 179 + exit 180 + (match cmd with 181 + | Post t -> post ~sw ~client agent t 182 + | Timeline n -> timeline ~sw ~client agent n 183 + | Profile a -> profile ~sw ~client agent a 184 + | Follow d -> follow ~sw ~client agent d 185 + | None -> 0)) 186 + | _ -> 187 + Printf.printf "Error: --user and --password required\n"; 188 + exit 1)
+14
examples/bsky_bot/dune
··· 1 + (executable 2 + (name bsky_bot) 3 + (public_name bsky_bot) 4 + (package atproto) 5 + (libraries 6 + atproto-api 7 + atproto-xrpc 8 + atproto-effects 9 + climate 10 + eio_main 11 + cohttp-eio 12 + tls-eio 13 + mirage-crypto-rng.unix 14 + uri))
+135
examples/feed_generator/README.md
··· 1 + # Feed Generator 2 + 3 + AT Protocol firehose client with filtering and feed skeleton generation. 4 + 5 + This example demonstrates how to: 6 + - Subscribe to the Bluesky network firehose using `atproto-sync` 7 + - Filter events by type (posts, likes, follows, etc.) 8 + - Filter posts by keyword content 9 + - Parse and decode CAR files and DAG-CBOR records 10 + - Generate feed skeleton output for custom feed generators 11 + 12 + ## Building 13 + 14 + ```bash 15 + dune build examples/feed_generator/feed_generator.exe 16 + ``` 17 + 18 + ## Usage 19 + 20 + ```bash 21 + dune exec examples/feed_generator/feed_generator.exe -- [OPTIONS] 22 + ``` 23 + 24 + ### Options 25 + 26 + | Option | Description | 27 + |--------|-------------| 28 + | `--cursor <INT64>` | Resume from a specific sequence number | 29 + | `--limit <INT>` | Stop after N matched events | 30 + | `--filter <TYPES>` | Filter by event type (comma-separated) | 31 + | `-k, --keyword <STRING>` | Filter posts containing keyword (case-insensitive) | 32 + | `--json` | Output events as JSON | 33 + | `--content` | Include decoded record content in output | 34 + | `-s, --skeleton` | Output feed skeleton at end (for custom feeds) | 35 + 36 + ### Event Types 37 + 38 + Available filter types: 39 + - `posts` - app.bsky.feed.post records 40 + - `likes` - app.bsky.feed.like records 41 + - `follows` - app.bsky.graph.follow records 42 + - `reposts` - app.bsky.feed.repost records 43 + - `blocks` - app.bsky.graph.block records 44 + - `lists` - app.bsky.graph.list records 45 + - `profiles` - app.bsky.actor.profile records 46 + - `feeds` - app.bsky.feed.generator records 47 + - `commits` - all commit events 48 + - `identities` - identity change events 49 + - `accounts` - account status events 50 + - `handles` - handle change events 51 + - `tombstones` - account deletion events 52 + 53 + ## Examples 54 + 55 + ### Watch all posts 56 + 57 + ```bash 58 + feed_generator --filter posts 59 + ``` 60 + 61 + ### Filter posts by keyword 62 + 63 + ```bash 64 + feed_generator --filter posts --keyword "ocaml" --limit 10 65 + ``` 66 + 67 + ### Generate a feed skeleton 68 + 69 + Collect posts matching a keyword and output a feed skeleton suitable for serving via `app.bsky.feed.getFeedSkeleton`: 70 + 71 + ```bash 72 + feed_generator --filter posts --keyword "rust" --limit 50 --skeleton 73 + ``` 74 + 75 + Output includes: 76 + ```json 77 + { 78 + "feed": [ 79 + {"post": "at://did:plc:xxx/app.bsky.feed.post/abc123"}, 80 + {"post": "at://did:plc:yyy/app.bsky.feed.post/def456"} 81 + ] 82 + } 83 + ``` 84 + 85 + ### JSON output with full record content 86 + 87 + ```bash 88 + feed_generator --filter posts --json --content --limit 5 89 + ``` 90 + 91 + ### Watch multiple event types 92 + 93 + ```bash 94 + feed_generator --filter posts,likes,follows --limit 100 95 + ``` 96 + 97 + ### Resume from a cursor 98 + 99 + ```bash 100 + feed_generator --cursor 1234567890 --limit 50 101 + ``` 102 + 103 + ### Watch all activity with decoded content 104 + 105 + ```bash 106 + feed_generator --content 107 + ``` 108 + 109 + ## Architecture 110 + 111 + The tool uses several AT Protocol libraries: 112 + 113 + - **atproto-sync**: Firehose subscription and event parsing 114 + - **atproto-ipld**: CAR file parsing, DAG-CBOR decoding, CID handling 115 + 116 + Key components: 117 + 1. **WebSocket client**: TLS-enabled WebSocket connection to the relay 118 + 2. **Event handler**: Processes commit, identity, account, and other events 119 + 3. **Type filter**: Matches operations against Lexicon record types 120 + 4. **Keyword filter**: Case-insensitive text search in post records 121 + 5. **Feed skeleton**: Collects AT URIs for matched posts 122 + 123 + ## Building a Custom Feed 124 + 125 + To build a complete custom feed generator: 126 + 127 + 1. Use this tool to prototype your filtering logic 128 + 2. Run with `--skeleton` to verify the output format 129 + 3. Implement a persistent service that: 130 + - Subscribes to the firehose continuously 131 + - Stores matched post URIs in a database 132 + - Serves the feed via XRPC endpoint `app.bsky.feed.getFeedSkeleton` 133 + - Publishes a feed generator record to your repository 134 + 135 + See the [AT Protocol Feed Generator documentation](https://docs.bsky.app/docs/starter-templates/custom-feeds) for more details.
+17
examples/feed_generator/dune
··· 1 + (executable 2 + (name feed_generator) 3 + (public_name feed_generator) 4 + (package atproto) 5 + (libraries 6 + atproto-sync 7 + atproto-ipld 8 + climate 9 + uri 10 + eio 11 + eio_main 12 + tls-eio 13 + ca-certs-nss 14 + mirage-crypto-rng.unix 15 + base64 16 + cstruct)) 17 +
-32
examples/firehose_demo/dune
··· 1 - (executable 2 - (name firehose_demo) 3 - (public_name firehose_demo) 4 - (package atproto) 5 - (libraries 6 - atproto-sync 7 - atproto-ipld 8 - climate 9 - uri 10 - eio 11 - eio_main 12 - tls-eio 13 - ca-certs-nss 14 - mirage-crypto-rng.unix 15 - base64 16 - cstruct)) 17 - 18 - (executable 19 - (name firehose_demo_lwt) 20 - (public_name firehose_demo_lwt) 21 - (package atproto) 22 - (libraries 23 - atproto-sync 24 - atproto-ipld 25 - climate 26 - uri 27 - lwt 28 - lwt.unix 29 - websocket-lwt-unix 30 - conduit-lwt-unix 31 - tls-lwt 32 - mirage-crypto-rng.unix))
+131 -11
examples/firehose_demo/firehose_demo.ml examples/feed_generator/feed_generator.ml
··· 1 - (** Firehose Demo - AT Protocol real-time events. Use --help for options. *) 1 + (** Feed Generator - AT Protocol firehose client with filtering. 2 + 3 + Subscribe to the Bluesky firehose, filter events by type or keyword, and 4 + optionally output a feed skeleton for custom feed generation. 5 + 6 + Use --help for options. *) 2 7 3 8 module Firehose = Atproto_sync.Firehose 4 9 module Car = Atproto_ipld.Car ··· 168 173 | Firehose.Tombstone _ -> List.mem Tombstones filters 169 174 | Firehose.Info _ | Firehose.StreamError _ -> true) 170 175 176 + (** {1 Keyword Filtering} *) 177 + 178 + let contains_keyword keyword text = 179 + let kw = String.lowercase_ascii keyword in 180 + let txt = String.lowercase_ascii text in 181 + let kw_len = String.length kw in 182 + let txt_len = String.length txt in 183 + if kw_len > txt_len then false 184 + else 185 + let rec search i = 186 + if i > txt_len - kw_len then false 187 + else if String.sub txt i kw_len = kw then true 188 + else search (i + 1) 189 + in 190 + search 0 191 + 192 + let text_matches_keyword keyword blocks (op : Firehose.operation) = 193 + match find_block blocks op with 194 + | Some cbor -> ( 195 + match get_text cbor with 196 + | Some text -> contains_keyword keyword text 197 + | None -> false) 198 + | None -> false 199 + 200 + let event_matches_keyword keyword event = 201 + match keyword with 202 + | None -> true 203 + | Some kw -> ( 204 + match event with 205 + | Firehose.Commit evt -> 206 + let blocks = extract_blocks evt.blocks in 207 + List.exists 208 + (fun op -> 209 + op.Firehose.action = `Create 210 + && prefix_match "app.bsky.feed.post" op.path 211 + && text_matches_keyword kw blocks op) 212 + evt.ops 213 + | _ -> false) 214 + 215 + (** {1 Feed Skeleton} *) 216 + 217 + let max_feed_size = 1000 218 + let feed_posts : string Queue.t = Queue.create () 219 + 220 + let add_to_feed uri = 221 + Queue.add uri feed_posts; 222 + if Queue.length feed_posts > max_feed_size then ignore (Queue.pop feed_posts) 223 + 224 + let collect_post_uris (evt : Firehose.commit_event) keyword = 225 + let blocks = extract_blocks evt.blocks in 226 + List.iter 227 + (fun (op : Firehose.operation) -> 228 + if 229 + op.action = `Create 230 + && prefix_match "app.bsky.feed.post" op.path 231 + && 232 + match keyword with 233 + | None -> true 234 + | Some kw -> text_matches_keyword kw blocks op 235 + then 236 + let rkey = 237 + match String.rindex_opt op.path '/' with 238 + | Some i -> String.sub op.path (i + 1) (String.length op.path - i - 1) 239 + | None -> op.path 240 + in 241 + let uri = 242 + Printf.sprintf "at://%s/app.bsky.feed.post/%s" evt.repo rkey 243 + in 244 + add_to_feed uri) 245 + evt.ops 246 + 247 + let print_feed_skeleton () = 248 + let items = Queue.fold (fun acc x -> x :: acc) [] feed_posts |> List.rev in 249 + if items = [] then () 250 + else begin 251 + Printf.printf "\n--- Feed Skeleton (app.bsky.feed.getFeedSkeleton) ---\n"; 252 + Printf.printf "{\n \"feed\": [\n"; 253 + List.iteri 254 + (fun i uri -> 255 + Printf.printf " {\"post\": \"%s\"}" uri; 256 + if i < List.length items - 1 then Printf.printf ","; 257 + Printf.printf "\n") 258 + items; 259 + Printf.printf " ]\n}\n" 260 + end 261 + 262 + (** {1 WebSocket Client} *) 263 + 171 264 module Ws = struct 172 265 type conn = { 173 266 socket : Tls_eio.t; ··· 413 506 | _ -> None); 414 507 } 415 508 509 + (** {1 Configuration} *) 510 + 416 511 type config = { 417 512 cursor : int64 option; 418 513 limit : int option; 419 514 filters : filter list; 515 + keyword : string option; 420 516 json : bool; 421 517 rich : bool; 518 + skeleton : bool; 422 519 } 423 520 424 521 let parse_filters s = ··· 438 535 let config_parser = 439 536 let open Climate.Arg_parser in 440 537 let+ cursor = named_opt [ "cursor" ] int64_conv ~doc:"Resume from sequence" 441 - and+ limit = named_opt [ "limit" ] int ~doc:"Stop after N events" 538 + and+ limit = named_opt [ "limit" ] int ~doc:"Stop after N matched events" 442 539 and+ filter_str = 443 540 named_opt [ "filter" ] string 444 - ~doc:"Filter (posts,likes,follows,reposts,etc.)" 541 + ~doc:"Filter by type (posts,likes,follows,reposts,etc.)" 542 + and+ keyword = 543 + named_opt [ "keyword"; "k" ] string ~doc:"Filter posts by keyword" 445 544 and+ json = flag [ "json" ] ~doc:"JSON output" 446 - and+ rich = flag [ "content" ] ~doc:"Show record content" in 545 + and+ rich = flag [ "content" ] ~doc:"Show record content" 546 + and+ skeleton = flag [ "skeleton"; "s" ] ~doc:"Output feed skeleton at end" in 447 547 { 448 548 cursor; 449 549 limit; 450 550 filters = Option.fold ~none:[] ~some:parse_filters filter_str; 551 + keyword; 451 552 json; 452 553 rich; 554 + skeleton; 453 555 } 454 556 455 557 let cli = 456 - Climate.Command.singleton ~doc:"AT Protocol firehose demo" config_parser 558 + Climate.Command.singleton 559 + ~doc:"AT Protocol firehose client and feed generator" config_parser 560 + 561 + (** {1 Stats} *) 457 562 458 563 type stats = { 459 564 mutable total : int; ··· 472 577 let stats = { total = 0; matched = 0; last_seq = 0L; start = 0. } 473 578 let interrupted = ref false 474 579 580 + (** {1 Main} *) 581 + 475 582 let run ~net ~sw config = 476 583 let uri = 477 584 Uri.of_string "wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos" ··· 489 596 (match Firehose.event_seq event with 490 597 | Some s -> stats.last_seq <- s 491 598 | None -> ()); 492 - if event_matches config.filters event then begin 599 + (* Collect posts for skeleton if enabled *) 600 + (if config.skeleton then 601 + match event with 602 + | Firehose.Commit evt -> collect_post_uris evt config.keyword 603 + | _ -> ()); 604 + (* Check filters *) 605 + let type_match = event_matches config.filters event in 606 + let keyword_match = event_matches_keyword config.keyword event in 607 + if type_match && keyword_match then begin 493 608 print_endline (fmt event); 494 609 stats.matched <- stats.matched + 1; 495 610 match config.limit with ··· 507 622 let () = 508 623 let config = 509 624 Climate.Command.run 510 - ~program_name:(Climate.Program_name.Literal "firehose_demo") cli 625 + ~program_name:(Climate.Program_name.Literal "feed_generator") cli 511 626 in 512 627 if not config.json then begin 513 - Printf.printf "AT Protocol Firehose Demo\n=========================\n\n%!"; 628 + Printf.printf "Feed Generator\n==============\n\n%!"; 514 629 if config.filters <> [] then 515 - Printf.printf "Filtering: %s\n\n%!" 516 - (String.concat ", " (List.map filter_to_string config.filters)) 630 + Printf.printf "Type filter: %s\n%!" 631 + (String.concat ", " (List.map filter_to_string config.filters)); 632 + (match config.keyword with 633 + | Some kw -> Printf.printf "Keyword: \"%s\"\n%!" kw 634 + | None -> ()); 635 + Printf.printf "\n%!" 517 636 end; 518 637 Sys.set_signal Sys.sigint (Sys.Signal_handle (fun _ -> interrupted := true)); 519 638 stats.start <- Unix.gettimeofday (); ··· 525 644 with 526 645 | Failure m -> Printf.eprintf "Error: %s\n%!" m 527 646 | e -> Printf.eprintf "Error: %s\n%!" (Printexc.to_string e)); 528 - if not config.json then print_stats stats 647 + if not config.json then print_stats stats; 648 + if config.skeleton then print_feed_skeleton ()
-452
examples/firehose_demo/firehose_demo_lwt.ml
··· 1 - (** Firehose Demo (Lwt version) - AT Protocol real-time events. Uses 2 - websocket-lwt-unix for WebSocket handling. Use --help for options. *) 3 - 4 - open Lwt.Infix 5 - module Firehose = Atproto_sync.Firehose 6 - module Car = Atproto_ipld.Car 7 - module Dag_cbor = Atproto_ipld.Dag_cbor 8 - module Dag_json = Atproto_ipld.Dag_json 9 - module Cid = Atproto_ipld.Cid 10 - 11 - (* Global interrupt handling via Lwt *) 12 - let interrupt_wakener : (unit, exn) result Lwt.u option ref = ref None 13 - let interrupted = ref false 14 - 15 - let trigger_interrupt () = 16 - interrupted := true; 17 - match !interrupt_wakener with 18 - | Some w -> 19 - Lwt.wakeup_later w (Error Exit); 20 - interrupt_wakener := None 21 - | None -> () 22 - 23 - let truncate n s = 24 - if String.length s <= n then s else String.sub s 0 (n - 3) ^ "..." 25 - 26 - let single_line s = 27 - String.map (fun c -> if c = '\n' || c = '\r' then ' ' else c) s 28 - 29 - let opt_or d = Option.value ~default:d 30 - 31 - let extract_blocks blocks_data = 32 - if String.length blocks_data = 0 then [] 33 - else 34 - match Car.read blocks_data with 35 - | Error _ -> [] 36 - | Ok (_, blocks) -> 37 - List.filter_map 38 - (fun (b : Car.block) -> 39 - match Dag_cbor.decode b.data with 40 - | Ok cbor -> Some (Cid.to_string b.cid, cbor) 41 - | _ -> None) 42 - blocks 43 - 44 - let find_block blocks (op : Firehose.operation) = 45 - match op.cid with 46 - | Some cid -> List.assoc_opt (Cid.to_string cid) blocks 47 - | None -> None 48 - 49 - let get_text cbor = 50 - match cbor with 51 - | Dag_cbor.Map pairs -> Firehose.get_string "text" pairs 52 - | _ -> None 53 - 54 - let format_op blocks (op : Firehose.operation) = 55 - let action = 56 - match op.action with `Create -> "+" | `Update -> "~" | `Delete -> "-" 57 - in 58 - let content = 59 - match find_block blocks op with 60 - | Some cbor -> ( 61 - match get_text cbor with 62 - | Some t -> " \"" ^ truncate 70 (single_line t) ^ "\"" 63 - | None -> "") 64 - | None -> "" 65 - in 66 - Printf.sprintf "%s%s%s" action op.path content 67 - 68 - let format_event ?(rich = false) = function 69 - | Firehose.Commit evt -> 70 - let blocks = if rich then extract_blocks evt.blocks else [] in 71 - let ops = 72 - String.concat 73 - (if rich then "\n " else " ") 74 - (List.map (format_op blocks) evt.ops) 75 - in 76 - Printf.sprintf "#commit seq=%Ld repo=%s%s%s" evt.seq evt.repo 77 - (if rich then "\n " else " ") 78 - ops 79 - | Firehose.Identity e -> 80 - Printf.sprintf "#identity seq=%Ld did=%s handle=%s" e.seq e.did 81 - (opt_or "-" e.handle) 82 - | Firehose.Account e -> 83 - Printf.sprintf "#account seq=%Ld did=%s active=%b" e.seq e.did e.active 84 - | Firehose.Handle e -> 85 - Printf.sprintf "#handle seq=%Ld did=%s handle=%s" e.seq e.did e.handle 86 - | Firehose.Tombstone e -> 87 - Printf.sprintf "#tombstone seq=%Ld did=%s" e.seq e.did 88 - | Firehose.Info m -> Printf.sprintf "#info name=%s" m.name 89 - | Firehose.StreamError err -> Printf.sprintf "#error %s" err 90 - 91 - let json_of_op_with_record blocks (op : Firehose.operation) = 92 - let base_json = Firehose.operation_to_json op in 93 - match find_block blocks op with 94 - | Some cbor -> 95 - let record_json = Yojson.Safe.from_string (Dag_json.encode_string cbor) in 96 - Firehose.add_record_to_op_json base_json record_json 97 - | None -> base_json 98 - 99 - let json_of_event ?(rich = false) event = 100 - let json = 101 - match event with 102 - | Firehose.Commit evt when rich -> ( 103 - let blocks = extract_blocks evt.blocks in 104 - let ops = `List (List.map (json_of_op_with_record blocks) evt.ops) in 105 - let base = Firehose.commit_event_to_json evt in 106 - match base with 107 - | `Assoc fields -> 108 - `Assoc 109 - (List.map 110 - (fun (k, v) -> if k = "ops" then (k, ops) else (k, v)) 111 - fields) 112 - | _ -> base) 113 - | _ -> Firehose.event_to_json event 114 - in 115 - Yojson.Safe.to_string json 116 - 117 - type filter = 118 - | Posts 119 - | Likes 120 - | Follows 121 - | Reposts 122 - | Blocks 123 - | Lists 124 - | Profiles 125 - | Feeds 126 - | Commits 127 - | Identities 128 - | Accounts 129 - | Handles 130 - | Tombstones 131 - 132 - let filters = 133 - [ 134 - ("posts", Posts); 135 - ("likes", Likes); 136 - ("follows", Follows); 137 - ("reposts", Reposts); 138 - ("blocks", Blocks); 139 - ("lists", Lists); 140 - ("profiles", Profiles); 141 - ("feeds", Feeds); 142 - ("commits", Commits); 143 - ("identities", Identities); 144 - ("accounts", Accounts); 145 - ("handles", Handles); 146 - ("tombstones", Tombstones); 147 - ] 148 - 149 - let filter_of_string s = List.assoc_opt s filters 150 - let filter_to_string f = fst (List.find (fun (_, v) -> v = f) filters) 151 - 152 - let prefix_match prefix path = 153 - String.length path >= String.length prefix 154 - && String.sub path 0 (String.length prefix) = prefix 155 - 156 - let op_matches filter (op : Firehose.operation) = 157 - match filter with 158 - | Posts -> prefix_match "app.bsky.feed.post" op.path 159 - | Likes -> prefix_match "app.bsky.feed.like" op.path 160 - | Follows -> prefix_match "app.bsky.graph.follow" op.path 161 - | Reposts -> prefix_match "app.bsky.feed.repost" op.path 162 - | Blocks -> prefix_match "app.bsky.graph.block" op.path 163 - | Lists -> prefix_match "app.bsky.graph.list" op.path 164 - | Profiles -> prefix_match "app.bsky.actor.profile" op.path 165 - | Feeds -> prefix_match "app.bsky.feed.generator" op.path 166 - | Commits -> true 167 - | _ -> false 168 - 169 - let event_matches filters event = 170 - match filters with 171 - | [] -> true 172 - | _ -> ( 173 - match event with 174 - | Firehose.Commit evt -> 175 - List.exists 176 - (fun f -> f = Commits || List.exists (op_matches f) evt.ops) 177 - filters 178 - | Firehose.Identity _ -> List.mem Identities filters 179 - | Firehose.Account _ -> List.mem Accounts filters 180 - | Firehose.Handle _ -> List.mem Handles filters 181 - | Firehose.Tombstone _ -> List.mem Tombstones filters 182 - | Firehose.Info _ | Firehose.StreamError _ -> true) 183 - 184 - (** WebSocket client using websocket-lwt-unix *) 185 - module Ws = struct 186 - type conn = Websocket_lwt_unix.conn 187 - 188 - let connect uri : (conn, string) result Lwt.t = 189 - let host = Uri.host uri |> Option.value ~default:"localhost" in 190 - let scheme = Uri.scheme uri |> Option.value ~default:"wss" in 191 - let port = 192 - Uri.port uri |> Option.value ~default:(if scheme = "wss" then 443 else 80) 193 - in 194 - let resource = Uri.path_and_query uri in 195 - (* Resolve hostname to IP *) 196 - Lwt.catch 197 - (fun () -> 198 - Lwt_unix.getaddrinfo host (string_of_int port) 199 - [ Unix.AI_SOCKTYPE Unix.SOCK_STREAM ] 200 - >>= function 201 - | [] -> Lwt.return (Error ("DNS failed: " ^ host)) 202 - | ai :: _ -> 203 - let ip = 204 - match ai.Unix.ai_addr with 205 - | Unix.ADDR_INET (addr, _) -> Ipaddr_unix.of_inet_addr addr 206 - | _ -> failwith "Unexpected address type" 207 - in 208 - let endp = `TLS_native (`Hostname host, `IP ip, `Port port) in 209 - Websocket_lwt_unix.connect endp (Uri.with_path uri resource) 210 - >|= fun conn -> Ok conn) 211 - (fun exn -> Lwt.return (Error (Printexc.to_string exn))) 212 - 213 - (* Buffer for accumulating fragmented messages *) 214 - let frag_buf = Buffer.create 65536 215 - let frag_opcode = ref Websocket.Frame.Opcode.Continuation 216 - 217 - let recv_impl conn : (string, string) result Lwt.t = 218 - let open Websocket in 219 - let rec read_binary () = 220 - Lwt.catch 221 - (fun () -> 222 - Websocket_lwt_unix.read conn >>= fun frame -> 223 - match frame.Frame.opcode with 224 - | Frame.Opcode.Binary -> 225 - if frame.Frame.final then 226 - (* Complete message in single frame *) 227 - Lwt.return (Ok frame.Frame.content) 228 - else begin 229 - (* Start of fragmented binary message *) 230 - Buffer.clear frag_buf; 231 - Buffer.add_string frag_buf frame.Frame.content; 232 - frag_opcode := Frame.Opcode.Binary; 233 - read_binary () 234 - end 235 - | Frame.Opcode.Continuation -> 236 - (* Continuation of fragmented message *) 237 - Buffer.add_string frag_buf frame.Frame.content; 238 - if frame.Frame.final then begin 239 - (* Fragmented message complete *) 240 - let data = Buffer.contents frag_buf in 241 - Buffer.clear frag_buf; 242 - if !frag_opcode = Frame.Opcode.Binary then Lwt.return (Ok data) 243 - else 244 - (* Was a text message, skip it *) 245 - read_binary () 246 - end 247 - else read_binary () 248 - | Frame.Opcode.Text -> 249 - if frame.Frame.final then 250 - (* Skip complete text frames *) 251 - read_binary () 252 - else begin 253 - (* Start of fragmented text message - track but skip *) 254 - Buffer.clear frag_buf; 255 - Buffer.add_string frag_buf frame.Frame.content; 256 - frag_opcode := Frame.Opcode.Text; 257 - read_binary () 258 - end 259 - | Frame.Opcode.Close -> Lwt.return (Error "Connection closed") 260 - | Frame.Opcode.Ping -> 261 - (* Respond to ping with pong *) 262 - let pong = 263 - Frame.create ~opcode:Frame.Opcode.Pong 264 - ~content:frame.Frame.content () 265 - in 266 - Websocket_lwt_unix.write conn pong >>= fun () -> read_binary () 267 - | Frame.Opcode.Pong -> read_binary () 268 - | _ -> read_binary ()) 269 - (fun exn -> Lwt.return (Error (Printexc.to_string exn))) 270 - in 271 - read_binary () 272 - 273 - (** Interruptible recv - races against interrupt signal *) 274 - let recv conn : (string, string) result Lwt.t = 275 - (* Create a waiter that can be woken by Ctrl+C *) 276 - let interrupt_waiter, wakener = Lwt.wait () in 277 - interrupt_wakener := Some wakener; 278 - (* Race between recv and interrupt *) 279 - Lwt.pick 280 - [ 281 - recv_impl conn; 282 - ( interrupt_waiter >>= function 283 - | Ok () -> Lwt.return (Error "Interrupted") 284 - | Error _ -> Lwt.return (Error "Interrupted") ); 285 - ] 286 - >>= fun result -> 287 - (* Clear the wakener if we won the race *) 288 - interrupt_wakener := None; 289 - Lwt.return result 290 - 291 - let close conn = 292 - Lwt.catch 293 - (fun () -> 294 - let close_frame = Websocket.Frame.close 1000 in 295 - Websocket_lwt_unix.write conn close_frame >>= fun () -> 296 - Websocket_lwt_unix.close_transport conn) 297 - (fun _ -> Lwt.return_unit) 298 - end 299 - 300 - (** Effect handler that uses Lwt for WebSocket operations *) 301 - let with_websocket_lwt f = 302 - let open Effect.Deep in 303 - (* We need to track Lwt state across effect invocations *) 304 - let ws_conn : Ws.conn option ref = ref None in 305 - try_with f () 306 - { 307 - effc = 308 - (fun (type a) (eff : a Effect.t) -> 309 - match eff with 310 - | Firehose.Ws_connect uri -> 311 - Some 312 - (fun (k : (a, _) continuation) -> 313 - let result = Lwt_main.run (Ws.connect uri) in 314 - (match result with 315 - | Ok conn -> ws_conn := Some conn 316 - | Error _ -> ()); 317 - continue k (Result.map Obj.magic result)) 318 - | Firehose.Ws_recv _ws -> 319 - Some 320 - (fun k -> 321 - match !ws_conn with 322 - | None -> continue k (Error "Not connected") 323 - | Some conn -> 324 - let result = Lwt_main.run (Ws.recv conn) in 325 - continue k result) 326 - | Firehose.Ws_close _ws -> 327 - Some 328 - (fun k -> 329 - (match !ws_conn with 330 - | None -> () 331 - | Some conn -> 332 - Lwt_main.run (Ws.close conn); 333 - ws_conn := None); 334 - continue k ()) 335 - | _ -> None); 336 - } 337 - 338 - type config = { 339 - cursor : int64 option; 340 - limit : int option; 341 - filters : filter list; 342 - json : bool; 343 - rich : bool; 344 - } 345 - 346 - let parse_filters s = 347 - String.split_on_char ',' s 348 - |> List.filter_map (fun p -> filter_of_string (String.trim p)) 349 - 350 - let int64_conv = 351 - let open Climate.Arg_parser in 352 - make_conv 353 - ~parse:(fun s -> 354 - match Int64.of_string_opt s with 355 - | Some n -> Ok n 356 - | None -> Error (`Msg "invalid integer")) 357 - ~print:(fun fmt n -> Format.fprintf fmt "%Ld" n) 358 - () 359 - 360 - let config_parser = 361 - let open Climate.Arg_parser in 362 - let+ cursor = named_opt [ "cursor" ] int64_conv ~doc:"Resume from sequence" 363 - and+ limit = named_opt [ "limit" ] int ~doc:"Stop after N events" 364 - and+ filter_str = 365 - named_opt [ "filter" ] string 366 - ~doc:"Filter (posts,likes,follows,reposts,etc.)" 367 - and+ json = flag [ "json" ] ~doc:"JSON output" 368 - and+ rich = flag [ "content" ] ~doc:"Show record content" in 369 - { 370 - cursor; 371 - limit; 372 - filters = Option.fold ~none:[] ~some:parse_filters filter_str; 373 - json; 374 - rich; 375 - } 376 - 377 - let cli = 378 - Climate.Command.singleton ~doc:"AT Protocol firehose demo (Lwt)" config_parser 379 - 380 - type stats = { 381 - mutable total : int; 382 - mutable matched : int; 383 - mutable last_seq : int64; 384 - mutable start : float; 385 - } 386 - 387 - let print_stats s = 388 - let elapsed = Unix.gettimeofday () -. s.start in 389 - let rate = if elapsed > 0. then float_of_int s.total /. elapsed else 0. in 390 - Printf.printf 391 - "\n--- Stats: %d events (%d matched) | seq=%Ld | %.1fs | %.1f evt/s ---\n%!" 392 - s.total s.matched s.last_seq elapsed rate 393 - 394 - let stats = { total = 0; matched = 0; last_seq = 0L; start = 0. } 395 - let json_mode = ref false 396 - 397 - let run config = 398 - let uri = 399 - Uri.of_string "wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos" 400 - in 401 - let cfg = Firehose.config ~uri ?cursor:config.cursor () in 402 - if not config.json then 403 - Printf.printf "Connecting to %s...\n%!" 404 - (Uri.to_string (Firehose.build_uri cfg)); 405 - let fmt = 406 - if config.json then json_of_event ~rich:config.rich 407 - else format_event ~rich:config.rich 408 - in 409 - let handler event = 410 - stats.total <- stats.total + 1; 411 - (match Firehose.event_seq event with 412 - | Some s -> stats.last_seq <- s 413 - | None -> ()); 414 - if event_matches config.filters event then begin 415 - print_endline (fmt event); 416 - stats.matched <- stats.matched + 1; 417 - match config.limit with 418 - | Some max when stats.matched >= max -> false 419 - | _ -> true (* Always continue - interrupt handled via WebSocket error *) 420 - end 421 - else true (* Always continue processing *) 422 - in 423 - with_websocket_lwt (fun () -> 424 - if not config.json then Printf.printf "Connected!\n\n%!"; 425 - match Firehose.subscribe cfg ~handler with 426 - | Ok () -> 427 - if not config.json then 428 - Printf.printf "Subscribe completed normally\n%!" 429 - | Error e -> 430 - if not config.json then 431 - Printf.printf "Subscribe ended: %s\n%!" (Firehose.error_to_string e)) 432 - 433 - let () = 434 - let config = 435 - Climate.Command.run 436 - ~program_name:(Climate.Program_name.Literal "firehose_demo_lwt") cli 437 - in 438 - json_mode := config.json; 439 - if not config.json then begin 440 - Printf.printf 441 - "AT Protocol Firehose Demo (Lwt)\n===============================\n\n%!"; 442 - if config.filters <> [] then 443 - Printf.printf "Filtering: %s\n\n%!" 444 - (String.concat ", " (List.map filter_to_string config.filters)); 445 - Printf.printf "Press Ctrl+C to stop and show stats.\n\n%!" 446 - end; 447 - (* Use Lwt_unix signal handling for proper integration with Lwt *) 448 - let _ = Lwt_unix.on_signal Sys.sigint (fun _ -> trigger_interrupt ()) in 449 - stats.start <- Unix.gettimeofday (); 450 - Mirage_crypto_rng_unix.use_default (); 451 - (try run config with _ -> ()); 452 - if not config.json then print_stats stats
+68
examples/identity_tool/README.md
··· 1 + # identity_tool 2 + 3 + AT Protocol identity lookup and verification tool. Resolves handles to DIDs, DIDs to documents, and verifies bidirectional identity bindings. 4 + 5 + ## Usage 6 + 7 + ``` 8 + identity_tool [OPTIONS] <IDENTIFIER> 9 + ``` 10 + 11 + Where `<IDENTIFIER>` is a handle (e.g., `jay.bsky.social`) or a DID (e.g., `did:plc:z72i7hdynmk6r22z27h6tvur`). 12 + 13 + ## Options 14 + 15 + | Option | Description | 16 + |--------|-------------| 17 + | `-H, --resolve-handle` | Resolve handle to DID | 18 + | `-d, --resolve-did` | Resolve DID to document | 19 + | `-v, --verify` | Verify bidirectional identity (default) | 20 + | `-h, --help` | Show help message | 21 + 22 + ## Examples 23 + 24 + Verify identity (default mode): 25 + 26 + ```bash 27 + dune exec examples/identity_tool/identity_tool.exe -- jay.bsky.social 28 + ``` 29 + 30 + Resolve a handle to its DID: 31 + 32 + ```bash 33 + dune exec examples/identity_tool/identity_tool.exe -- -H jay.bsky.social 34 + ``` 35 + 36 + Resolve a DID to its document: 37 + 38 + ```bash 39 + dune exec examples/identity_tool/identity_tool.exe -- -d did:plc:z72i7hdynmk6r22z27h6tvur 40 + ``` 41 + 42 + ## Output 43 + 44 + Handle resolution shows the DID: 45 + 46 + ``` 47 + Handle: jay.bsky.social 48 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 49 + ``` 50 + 51 + DID resolution shows the full document: 52 + 53 + ``` 54 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 55 + Handle: at://jay.bsky.social 56 + Key: zQ3shXjHeiBuRCKmM... (multikey) 57 + Service: AtprotoPersonalDataServer -> https://morel.us-east.host.bsky.network 58 + ``` 59 + 60 + Verification confirms bidirectional binding: 61 + 62 + ``` 63 + PASSED 64 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 65 + Handle: jay.bsky.social 66 + Key: zQ3shXjHeiBuRCKmM... 67 + PDS: https://morel.us-east.host.bsky.network 68 + ```
+13
examples/identity_tool/dune
··· 1 + (executable 2 + (name identity_tool) 3 + (public_name identity_tool) 4 + (package atproto) 5 + (libraries 6 + atproto-identity 7 + atproto-syntax 8 + climate 9 + eio_main 10 + cohttp-eio 11 + tls-eio 12 + mirage-crypto-rng.unix 13 + uri))
+167
examples/identity_tool/identity_tool.ml
··· 1 + (** Identity Tool - Resolve handles/DIDs and verify AT Protocol identities. 2 + Usage: identity_tool [--resolve-handle|--resolve-did|--verify] <identifier> 3 + *) 4 + 5 + open Atproto_syntax 6 + open Atproto_identity 7 + 8 + (** {1 HTTP Client with cohttp-eio} *) 9 + 10 + let http_get ~sw ~client uri = 11 + try 12 + let resp, resp_body = Cohttp_eio.Client.call ~sw client `GET uri in 13 + let status = Cohttp.Response.status resp |> Cohttp.Code.code_of_status in 14 + let body = 15 + Eio.Buf_read.(of_flow ~max_size:(10 * 1024 * 1024) resp_body |> take_all) 16 + in 17 + Did_resolver.{ status; body } 18 + with e -> Did_resolver.{ status = 0; body = Printexc.to_string e } 19 + 20 + (** {1 Effect Handler} *) 21 + 22 + let run_with_eio ~sw ~client f = 23 + Effect.Deep.try_with f () 24 + { 25 + effc = 26 + (fun (type a) (eff : a Effect.t) -> 27 + match eff with 28 + | Did_resolver.Http_get uri -> 29 + Some 30 + (fun (k : (a, _) Effect.Deep.continuation) -> 31 + Effect.Deep.continue k (http_get ~sw ~client uri)) 32 + | Handle_resolver.Http_get uri -> 33 + Some 34 + (fun k -> 35 + let r = http_get ~sw ~client uri in 36 + Effect.Deep.continue k 37 + Handle_resolver.{ status = r.status; body = r.body }) 38 + | Handle_resolver.Dns_txt _ -> 39 + Some 40 + (fun k -> Effect.Deep.continue k Handle_resolver.Dns_not_found) 41 + | _ -> None); 42 + } 43 + 44 + (** {1 Commands} *) 45 + 46 + let resolve_handle ~sw ~client h = 47 + match Handle.of_string h with 48 + | Error _ -> 49 + Printf.printf "Error: Invalid handle\n"; 50 + 1 51 + | Ok handle -> 52 + run_with_eio ~sw ~client (fun () -> 53 + match Handle_resolver.resolve handle with 54 + | Error e -> 55 + Printf.printf "Error: %s\n" (Handle_resolver.error_to_string e); 56 + 1 57 + | Ok did -> 58 + Printf.printf "Handle: %s\nDID: %s\n" (Handle.to_string handle) 59 + (Did.to_string did); 60 + 0) 61 + 62 + let resolve_did ~sw ~client d = 63 + match Did.of_string d with 64 + | Error _ -> 65 + Printf.printf "Error: Invalid DID\n"; 66 + 1 67 + | Ok did -> 68 + run_with_eio ~sw ~client (fun () -> 69 + match Did_resolver.resolve_did did with 70 + | Error e -> 71 + Printf.printf "Error: %s\n" (Did_resolver.error_to_string e); 72 + 1 73 + | Ok doc -> 74 + Printf.printf "DID: %s\n" doc.id; 75 + List.iter 76 + (fun a -> Printf.printf "Handle: %s\n" a) 77 + doc.also_known_as; 78 + List.iter 79 + (fun (v : Did_resolver.verification_method) -> 80 + Printf.printf "Key: %s (%s)\n" 81 + (Option.value ~default:"-" v.public_key_multibase) 82 + v.type_) 83 + doc.verification_method; 84 + List.iter 85 + (fun (s : Did_resolver.service) -> 86 + Printf.printf "Service: %s -> %s\n" s.type_ s.service_endpoint) 87 + doc.service; 88 + 0) 89 + 90 + let verify ~sw ~client id = 91 + let is_did = String.length id > 4 && String.sub id 0 4 = "did:" in 92 + run_with_eio ~sw ~client (fun () -> 93 + let result = 94 + if is_did then 95 + match Did.of_string id with 96 + | Error _ -> Error "Invalid DID" 97 + | Ok d -> 98 + Identity.verify_did d |> Result.map_error Identity.error_to_string 99 + else 100 + match Handle.of_string id with 101 + | Error _ -> Error "Invalid handle" 102 + | Ok h -> 103 + Identity.verify_handle h 104 + |> Result.map_error Identity.error_to_string 105 + in 106 + match result with 107 + | Error e -> 108 + Printf.printf "FAILED: %s\n" e; 109 + 1 110 + | Ok v -> 111 + Printf.printf "PASSED\nDID: %s\nHandle: %s\n" (Did.to_string v.did) 112 + (Handle.to_string v.handle); 113 + Option.iter (Printf.printf "Key: %s\n") v.signing_key; 114 + Option.iter 115 + (fun u -> Printf.printf "PDS: %s\n" (Uri.to_string u)) 116 + v.pds_endpoint; 117 + 0) 118 + 119 + (** {1 CLI} *) 120 + 121 + type mode = Resolve_handle | Resolve_did | Verify 122 + 123 + let cli = 124 + let open Climate.Arg_parser in 125 + let+ id = 126 + pos_req 0 string ~value_name:"IDENTIFIER" ~doc:"Handle or DID to look up" 127 + and+ rh = flag [ "H"; "resolve-handle" ] ~doc:"Resolve handle to DID" 128 + and+ rd = flag [ "d"; "resolve-did" ] ~doc:"Resolve DID to document" 129 + and+ _vf = flag [ "v"; "verify" ] ~doc:"Verify bidirectional identity" in 130 + let mode = 131 + if rh then Resolve_handle else if rd then Resolve_did else Verify 132 + in 133 + (mode, id) 134 + 135 + let () = 136 + Mirage_crypto_rng_unix.use_default (); 137 + Eio_main.run @@ fun env -> 138 + Eio.Switch.run @@ fun sw -> 139 + let https_config = 140 + match 141 + Tls.Config.client ~authenticator:(fun ?ip:_ ~host:_ _ -> Ok None) () 142 + with 143 + | Ok c -> c 144 + | Error (`Msg m) -> failwith m 145 + in 146 + let https uri socket = 147 + let tls_host = 148 + match Uri.host uri with 149 + | Some h -> ( 150 + match Domain_name.of_string h with 151 + | Error _ -> None 152 + | Ok dn -> Domain_name.host dn |> Result.to_option) 153 + | None -> None 154 + in 155 + Tls_eio.client_of_flow https_config ?host:tls_host socket 156 + in 157 + let client = Cohttp_eio.Client.make ~https:(Some https) env#net in 158 + let mode, id = 159 + Climate.Command.run 160 + ~program_name:(Climate.Program_name.Literal "identity_tool") 161 + (Climate.Command.singleton ~doc:"AT Protocol identity lookup tool" cli) 162 + in 163 + exit 164 + (match mode with 165 + | Resolve_handle -> resolve_handle ~sw ~client id 166 + | Resolve_did -> resolve_did ~sw ~client id 167 + | Verify -> verify ~sw ~client id)
+138
examples/repo_inspector/README.md
··· 1 + # repo_inspector 2 + 3 + AT Protocol repository inspector for exploring repository contents, collections, and records. 4 + 5 + ## Usage 6 + 7 + ``` 8 + repo_inspector [OPTIONS] <DID> 9 + ``` 10 + 11 + Downloads and parses the repository for the given DID. The tool automatically resolves the user's PDS (Personal Data Server) from their DID document. 12 + 13 + ## Options 14 + 15 + | Option | Description | 16 + |--------|-------------| 17 + | `--pds <URL>` | PDS URL (auto-resolved from DID document if not specified) | 18 + | `-c, --collections` | List all collections with record counts | 19 + | `-r, --records` | Show individual records | 20 + | `--collection <STRING>` | Filter records by collection prefix | 21 + | `-n, --limit <INT>` | Limit number of records shown (default: 20) | 22 + | `-v, --verify` | Show commit signature information | 23 + | `-h, --help` | Show help message | 24 + 25 + ## Examples 26 + 27 + Show repository summary: 28 + 29 + ```bash 30 + dune exec examples/repo_inspector/repo_inspector.exe -- did:plc:z72i7hdynmk6r22z27h6tvur 31 + ``` 32 + 33 + List collections: 34 + 35 + ```bash 36 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 37 + did:plc:z72i7hdynmk6r22z27h6tvur --collections 38 + ``` 39 + 40 + Show records: 41 + 42 + ```bash 43 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 44 + did:plc:z72i7hdynmk6r22z27h6tvur --records --limit 10 45 + ``` 46 + 47 + Show only posts: 48 + 49 + ```bash 50 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 51 + did:plc:z72i7hdynmk6r22z27h6tvur --records --collection app.bsky.feed.post 52 + ``` 53 + 54 + Verify commit signature: 55 + 56 + ```bash 57 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 58 + did:plc:z72i7hdynmk6r22z27h6tvur --verify 59 + ``` 60 + 61 + Use a specific PDS (skip auto-resolution): 62 + 63 + ```bash 64 + dune exec examples/repo_inspector/repo_inspector.exe -- \ 65 + did:plc:z72i7hdynmk6r22z27h6tvur --pds https://puffball.us-east.host.bsky.network 66 + ``` 67 + 68 + ## Output 69 + 70 + Summary output: 71 + 72 + ``` 73 + Repository Inspector 74 + ==================== 75 + 76 + Resolving PDS for did:plc:z72i7hdynmk6r22z27h6tvur... 77 + PDS: https://puffball.us-east.host.bsky.network 78 + 79 + Fetching https://puffball.us-east.host.bsky.network/xrpc/com.atproto.sync.getRepo?did=... 80 + 81 + CAR Header 82 + ---------- 83 + Version: 1 84 + Roots: 1 85 + - bafyreib... 86 + Blocks: 1234 87 + 88 + Commit 89 + ------ 90 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 91 + Version: 3 92 + Rev: 3abc123def456 93 + Data: bafyreic... 94 + 95 + Collections 96 + ----------- 97 + app.bsky.feed.post: 456 98 + app.bsky.feed.like: 789 99 + app.bsky.graph.follow: 123 100 + app.bsky.actor.profile: 1 101 + 102 + Total records: 1369 103 + ``` 104 + 105 + Records output: 106 + 107 + ``` 108 + Records 109 + ------- 110 + app.bsky.feed.post/3abc123 111 + CID: bafyreig... 112 + Text: Hello world, this is my first post... 113 + 114 + app.bsky.feed.post/3def456 115 + CID: bafyreih... 116 + Text: Another post here... 117 + 118 + ... and 454 more 119 + ``` 120 + 121 + Verify output: 122 + 123 + ``` 124 + Commit Verification 125 + ------------------- 126 + DID: did:plc:z72i7hdynmk6r22z27h6tvur 127 + Signature: MEUCIQDa7H8c9X... 128 + Status: Signature present but key resolution not implemented 129 + (would need to resolve DID document to verify) 130 + ``` 131 + 132 + ## Notes 133 + 134 + - Automatically resolves the user's PDS from their DID document via PLC directory 135 + - Use `--pds` to skip auto-resolution and specify a PDS directly 136 + - Large repositories may take time to download 137 + - Record text preview is truncated to 60 characters 138 + - The `--collection` filter matches the start of the collection path
+19
examples/repo_inspector/dune
··· 1 + (executable 2 + (name repo_inspector) 3 + (public_name repo_inspector) 4 + (package atproto) 5 + (libraries 6 + atproto-repo 7 + atproto-mst 8 + atproto-ipld 9 + atproto-identity 10 + atproto-syntax 11 + atproto-crypto 12 + climate 13 + eio_main 14 + cohttp-eio 15 + tls-eio 16 + ca-certs-nss 17 + base64 18 + mirage-crypto-rng.unix 19 + uri))
+297
examples/repo_inspector/repo_inspector.ml
··· 1 + (** Repository Inspector - Explore AT Protocol repositories. 2 + 3 + This example demonstrates: 1. Downloading a repository as CAR file 2. 4 + Parsing the MST structure 3. Displaying records with CIDs 4. Verifying 5 + commit signatures 6 + 7 + Usage: repo_inspector <DID> [--pds URL] [--collections|--records|--verify] 8 + *) 9 + 10 + module Car = Atproto_ipld.Car 11 + module Dag_cbor = Atproto_ipld.Dag_cbor 12 + module Dag_json = Atproto_ipld.Dag_json 13 + module Cid = Atproto_ipld.Cid 14 + module Repo = Atproto_repo.Repo 15 + module Commit = Atproto_repo.Commit 16 + module Blockstore = Atproto_mst.Memory_blockstore 17 + module Mst = Atproto_mst.Make (Blockstore) 18 + module Did = Atproto_syntax.Did 19 + module Did_resolver = Atproto_identity.Did_resolver 20 + 21 + (** {1 HTTP Client with cohttp-eio} *) 22 + 23 + let http_get ~sw ~client uri = 24 + try 25 + let resp, resp_body = Cohttp_eio.Client.call ~sw client `GET uri in 26 + let status = Cohttp.Response.status resp |> Cohttp.Code.code_of_status in 27 + let body = 28 + Eio.Buf_read.(of_flow ~max_size:(100 * 1024 * 1024) resp_body |> take_all) 29 + in 30 + (status, body) 31 + with e -> (0, Printexc.to_string e) 32 + 33 + (** {1 DID Resolution Effect Handler} *) 34 + 35 + let run_with_resolver ~sw ~client f = 36 + Effect.Deep.try_with f () 37 + { 38 + effc = 39 + (fun (type a) (eff : a Effect.t) -> 40 + match eff with 41 + | Did_resolver.Http_get uri -> 42 + Some 43 + (fun (k : (a, _) Effect.Deep.continuation) -> 44 + let status, body = http_get ~sw ~client uri in 45 + Effect.Deep.continue k Did_resolver.{ status; body }) 46 + | _ -> None); 47 + } 48 + 49 + (** {1 PDS Resolution} *) 50 + 51 + let resolve_pds ~sw ~client did_str = 52 + match Did.of_string did_str with 53 + | Error _ -> Error "Invalid DID format" 54 + | Ok did -> 55 + run_with_resolver ~sw ~client (fun () -> 56 + match Did_resolver.resolve_did did with 57 + | Error e -> Error (Did_resolver.error_to_string e) 58 + | Ok doc -> ( 59 + (* Find AtprotoPersonalDataServer service *) 60 + match 61 + List.find_opt 62 + (fun (s : Did_resolver.service) -> 63 + s.type_ = "AtprotoPersonalDataServer") 64 + doc.service 65 + with 66 + | Some s -> Ok s.service_endpoint 67 + | None -> Error "No PDS service found in DID document")) 68 + 69 + (** {1 Repository Fetching} *) 70 + 71 + let fetch_repo ~sw ~client ~pds_url did = 72 + let uri = 73 + Uri.of_string (pds_url ^ "/xrpc/com.atproto.sync.getRepo") 74 + |> Fun.flip Uri.add_query_param ("did", [ did ]) 75 + in 76 + Printf.printf "Fetching %s...\n%!" (Uri.to_string uri); 77 + let status, body = http_get ~sw ~client uri in 78 + if status = 200 then Ok body 79 + else 80 + Error 81 + (Printf.sprintf "HTTP %d: %s" status 82 + (String.sub body 0 (min 200 (String.length body)))) 83 + 84 + (** {1 Repository Analysis} *) 85 + 86 + let truncate n s = 87 + if String.length s <= n then s else String.sub s 0 (n - 3) ^ "..." 88 + 89 + let single_line s = 90 + String.map (fun c -> if c = '\n' || c = '\r' then ' ' else c) s 91 + 92 + let get_text cbor = 93 + match cbor with 94 + | Dag_cbor.Map pairs -> ( 95 + match List.assoc_opt "text" pairs with 96 + | Some (Dag_cbor.String s) -> Some s 97 + | _ -> None) 98 + | _ -> None 99 + 100 + let analyze_repo car_data = 101 + match Car.read car_data with 102 + | Error e -> 103 + Printf.printf "Failed to parse CAR: %s\n" (Car.error_to_string e); 104 + None 105 + | Ok (header, blocks) -> ( 106 + Printf.printf "\nCAR Header\n----------\n"; 107 + Printf.printf "Version: %d\n" header.version; 108 + Printf.printf "Roots: %d\n" (List.length header.roots); 109 + List.iter 110 + (fun r -> Printf.printf " - %s\n" (Cid.to_string r)) 111 + header.roots; 112 + Printf.printf "Blocks: %d\n\n" (List.length blocks); 113 + (* Build blockstore *) 114 + let store = Blockstore.create () in 115 + List.iter 116 + (fun (b : Car.block) -> Blockstore.put store b.cid b.data) 117 + blocks; 118 + (* Find commit *) 119 + let commit_cid = List.hd header.roots in 120 + let commit_data = Blockstore.get store commit_cid in 121 + match commit_data with 122 + | None -> 123 + Printf.printf "Commit block not found\n"; 124 + None 125 + | Some data -> ( 126 + match Commit.of_dag_cbor data with 127 + | Error _ -> 128 + Printf.printf "Failed to parse commit\n"; 129 + None 130 + | Ok commit -> Some (store, commit))) 131 + 132 + let show_commit (commit : Commit.t) = 133 + Printf.printf "Commit\n------\n"; 134 + Printf.printf "DID: %s\n" commit.did; 135 + Printf.printf "Version: %d\n" commit.version; 136 + Printf.printf "Rev: %s\n" commit.rev; 137 + Printf.printf "Data: %s\n" (Cid.to_string commit.data); 138 + (match commit.prev with 139 + | Some p -> Printf.printf "Prev: %s\n" (Cid.to_string p) 140 + | None -> ()); 141 + Printf.printf "\n" 142 + 143 + let show_collections store (commit : Commit.t) = 144 + Printf.printf "Collections\n-----------\n"; 145 + let entries = Mst.to_list store commit.data in 146 + let collections = Hashtbl.create 16 in 147 + List.iter 148 + (fun (key, _) -> 149 + match String.index_opt key '/' with 150 + | Some i -> 151 + let coll = String.sub key 0 i in 152 + let count = 153 + match Hashtbl.find_opt collections coll with 154 + | Some n -> n + 1 155 + | None -> 1 156 + in 157 + Hashtbl.replace collections coll count 158 + | None -> ()) 159 + entries; 160 + Hashtbl.iter 161 + (fun coll count -> Printf.printf " %s: %d\n" coll count) 162 + collections; 163 + Printf.printf "\nTotal records: %d\n" (List.length entries) 164 + 165 + let show_records ?(limit = 20) ?(collection = None) store (commit : Commit.t) = 166 + Printf.printf "Records%s\n-------\n" 167 + (match collection with Some c -> " (" ^ c ^ ")" | None -> ""); 168 + let entries = Mst.to_list store commit.data in 169 + let filtered = 170 + match collection with 171 + | None -> entries 172 + | Some prefix -> 173 + List.filter 174 + (fun (key, _) -> 175 + String.length key > String.length prefix 176 + && String.sub key 0 (String.length prefix) = prefix) 177 + entries 178 + in 179 + let shown = ref 0 in 180 + List.iter 181 + (fun (key, cid) -> 182 + if !shown < limit then begin 183 + Printf.printf "%s\n CID: %s\n" key (Cid.to_string cid); 184 + (* Try to get and show content preview *) 185 + (match Blockstore.get store cid with 186 + | Some data -> ( 187 + match Dag_cbor.decode data with 188 + | Ok cbor -> ( 189 + match get_text cbor with 190 + | Some t -> 191 + Printf.printf " Text: %s\n" (truncate 60 (single_line t)) 192 + | None -> ()) 193 + | Error _ -> ()) 194 + | None -> ()); 195 + incr shown 196 + end) 197 + filtered; 198 + if List.length filtered > limit then 199 + Printf.printf "\n... and %d more\n" (List.length filtered - limit) 200 + 201 + let verify_commit _store (commit : Commit.t) = 202 + Printf.printf "Commit Verification\n-------------------\n"; 203 + Printf.printf "DID: %s\n" commit.did; 204 + let sig_b64 = 205 + if String.length commit.sig_ > 0 then 206 + truncate 40 (Base64.encode_exn commit.sig_) 207 + else "(none)" 208 + in 209 + Printf.printf "Signature: %s\n" sig_b64; 210 + (* Get signing key from DID - would need identity resolution *) 211 + Printf.printf 212 + "Status: Signature present but key resolution not implemented\n"; 213 + Printf.printf " (would need to resolve DID document to verify)\n" 214 + 215 + (** {1 CLI} *) 216 + 217 + type mode = Summary | Collections | Records of string option | Verify 218 + 219 + let cli = 220 + let open Climate.Arg_parser in 221 + let+ did = pos_req 0 string ~value_name:"DID" ~doc:"Repository DID to inspect" 222 + and+ pds = 223 + named_opt [ "pds" ] string ~doc:"PDS URL (auto-resolved if not specified)" 224 + and+ collections = flag [ "c"; "collections" ] ~doc:"List collections" 225 + and+ records = flag [ "r"; "records" ] ~doc:"Show records" 226 + and+ collection = 227 + named_opt [ "collection" ] string ~doc:"Filter by collection" 228 + and+ verify = flag [ "v"; "verify" ] ~doc:"Verify commit signature" 229 + and+ limit = named_opt [ "n"; "limit" ] int ~doc:"Limit records shown" in 230 + let mode = 231 + if collections then Collections 232 + else if records then Records collection 233 + else if verify then Verify 234 + else Summary 235 + in 236 + (did, pds, mode, Option.value limit ~default:20) 237 + 238 + let () = 239 + Mirage_crypto_rng_unix.use_default (); 240 + Eio_main.run @@ fun env -> 241 + Eio.Switch.run @@ fun sw -> 242 + let auth = 243 + match Ca_certs_nss.authenticator () with 244 + | Ok a -> a 245 + | Error (`Msg m) -> failwith m 246 + in 247 + let https_config = 248 + match Tls.Config.client ~authenticator:auth () with 249 + | Ok c -> c 250 + | Error (`Msg m) -> failwith m 251 + in 252 + let https uri socket = 253 + let tls_host = 254 + match Uri.host uri with 255 + | Some h -> ( 256 + match Domain_name.of_string h with 257 + | Error _ -> None 258 + | Ok dn -> Domain_name.host dn |> Result.to_option) 259 + | None -> None 260 + in 261 + Tls_eio.client_of_flow https_config ?host:tls_host socket 262 + in 263 + let client = Cohttp_eio.Client.make ~https:(Some https) env#net in 264 + let did, pds_opt, mode, limit = 265 + Climate.Command.run 266 + ~program_name:(Climate.Program_name.Literal "repo_inspector") 267 + (Climate.Command.singleton ~doc:"AT Protocol repository inspector" cli) 268 + in 269 + Printf.printf "Repository Inspector\n====================\n\n"; 270 + (* Resolve PDS if not provided *) 271 + let pds_url = 272 + match pds_opt with 273 + | Some url -> url 274 + | None -> ( 275 + Printf.printf "Resolving PDS for %s...\n%!" did; 276 + match resolve_pds ~sw ~client did with 277 + | Ok url -> 278 + Printf.printf "PDS: %s\n\n%!" url; 279 + url 280 + | Error e -> 281 + Printf.printf "Error resolving PDS: %s\n" e; 282 + exit 1) 283 + in 284 + match fetch_repo ~sw ~client ~pds_url did with 285 + | Error e -> 286 + Printf.printf "Error: %s\n" e; 287 + exit 1 288 + | Ok car_data -> ( 289 + match analyze_repo car_data with 290 + | None -> exit 1 291 + | Some (store, commit) -> ( 292 + show_commit commit; 293 + match mode with 294 + | Summary -> show_collections store commit 295 + | Collections -> show_collections store commit 296 + | Records coll -> show_records ~limit ~collection:coll store commit 297 + | Verify -> verify_commit store commit))