commits
- Bump version to 0.1.2
- Remove examples from atproto package (fix missing cohttp-eio error)
- Migrate all examples from cohttp-eio to hcs library:
- bsky_bot: use Hcs.Client for HTTP requests
- identity_tool: use Hcs.Client for HTTP requests
- repo_inspector: use Hcs.Client with 100MB response limit
- feed_generator: use Hcs.Websocket for firehose connection
- Update dune files to use hcs instead of cohttp-eio/tls-eio
- All 311 tests pass
- Add version 0.1.0 to dune-project
- Update OCaml requirement from >= 5.1 to >= 5.4 (tested version)
- Add all 13 sub-packages as dependencies of umbrella atproto package
- Regenerate all .opam files
- Remove all yojson dependencies from dune-project (fully migrated to atproto-json)
- Add atproto-json dependency to packages that need JSON (lexicon, identity, api)
- mirage-crypto-ec and mirage-crypto-rng already require >= 2.0.2
Note: mirage-crypto-rng-eio 1.2.0 pins all mirage-crypto packages to 1.2.0
which has a P-256 signature verification bug. Do not install it.
- Replace Yojson with Atproto_json (simdjson-backed) in all test files:
- test/mst/test_mst.ml
- test/crypto/test_crypto.ml
- test/sync/test_sync.ml
- test/ipld/test_ipld.ml
- test/compliance/compliance_report.ml
- test/compliance/run_compliance.ml
- Remove yojson dependency from all test dune files
- Add encode_pretty function to lib/json/atproto_json.ml for formatted JSON output
- Update test/lexicon/dune to remove stale yojson dependency
All tests pass except pre-existing crypto signature verification failures.
- Add Atproto_json wrapper around Simdjsont with helpers
- Migrate lib/xrpc/{client,oauth,server}.ml
- Migrate lib/lexicon/{parser,validator,codegen}.ml
- Migrate lib/crypto/jwt.ml
- Migrate lib/ipld/{dag_json,blob}.ml
- Migrate lib/identity/did_resolver.ml
- Migrate lib/sync/firehose.ml
- Migrate lib/api/{agent,richtext}.ml
- Download 315 official AT Protocol lexicon schemas
- Update dune files to use atproto-json package
All lib/ code now uses Atproto_json exclusively.
No direct Simdjsont or Yojson usage in lib/ (except Atproto_json impl).
Tests use Yojson only for fixture loading - not production code.
Add four example tools demonstrating AT Protocol library usage:
- bsky_bot: Bluesky bot for posting, timeline, profiles, following
- feed_generator: Firehose client with filtering and feed skeleton
(evolved from firehose_demo with keyword filtering)
- identity_tool: Handle/DID resolution and identity verification
- repo_inspector: Repository exploration with auto PDS resolution
Key improvements:
- repo_inspector now auto-resolves PDS from DID document
- Added --pds option to skip auto-resolution
- Fixed URL encoding issue (? was incorrectly encoded as %3F)
Add GUIDE.md showing complete workflow: firehose -> identity -> repo
Remove deprecated firehose_demo in favor of feed_generator.
The Lwt_main.run call blocks and doesn't respond to signals by itself.
This fix uses Lwt.pick to race the WebSocket recv against an interrupt
waiter that gets woken when SIGINT is received.
- Add trigger_interrupt() that wakes up the interrupt waiter
- recv() now races recv_impl() against the interrupt signal
- Signal handler calls trigger_interrupt() instead of just setting flag
Now pressing Ctrl+C properly stops the demo and shows statistics.
The websocket-lwt-unix library delivers individual WebSocket frames,
not complete messages. Large messages can be split across multiple
frames (Binary/Text followed by Continuation frames until final=true).
This fix:
- Adds frag_buf buffer to accumulate fragmented message data
- Tracks message type (Binary/Text) via frag_opcode
- Only returns complete binary messages when final frame received
- Properly handles fragmented text messages (skipped)
Without this fix, partial binary data was passed to the CBOR decoder
causing 'invalid payload CBOR' errors after ~1-2 seconds of operation.
- Run indefinitely until Ctrl+C, then print stats and exit cleanly
- Use Lwt_unix.on_signal for proper signal handling integration
- Suppress error messages on graceful exit (decode errors are expected)
- Check interrupted flag before and after WebSocket reads
- Add 'Press Ctrl+C to stop' message at startup
- Uses websocket-lwt-unix library for WebSocket handling
- Implements Firehose effects with Lwt_main.run in handlers
- 380 lines vs 528 for Eio version (28% reduction)
- Same CLI options: --filter, --limit, --json, --content, --cursor
- Use Dag_json.encode_string for record serialization instead of custom code
- Add Climate for CLI parsing with auto-generated help
- Add compact stats with SIGINT handling (prints on Ctrl+C)
- Remove custom record_content type and JSON escaping
- Simplify filter names to single form only
Line count: 805 → 547 (32% reduction)
Major architectural change:
- Use Firehose.subscribe from the library instead of manual event loop
- Move WebSocket implementation into Ws_handler module as effect handler
- Demo now focuses on filtering/formatting, not protocol details
The demo provides effect handlers for:
- Firehose.Ws_connect: TLS connection with WebSocket upgrade
- Firehose.Ws_recv: Frame parsing with fragmentation support
- Firehose.Ws_close: Connection cleanup
This separates concerns:
- Library: protocol logic (subscribe, decode_frame, event types)
- Demo: presentation (filtering, formatting, statistics)
- Ws_handler: I/O implementation (TLS, WebSocket framing)
Line count: 795 (down from 1204, -34%)
Major changes:
- Remove demo mode and sample data (~143 lines)
- Consolidate JSON formatting with shared helpers (json_opt, json_str)
- Merge duplicate formatting functions (format_event, format_event_json now take ~rich param)
- Simplify filter matching with prefix_match helper
- Compact statistics printing
- Remove verbose section comment banners
- Use more concise OCaml idioms throughout
All features preserved:
- --filter for event type filtering
- --json for JSON lines output
- --content for record content extraction
- --cursor for resuming from sequence
- --limit for stopping after N events
Remove duplicate CBOR helper functions and use Firehose module's
get_string, get_array, etc. instead of local reimplementations.
Changes:
- Remove get_cbor_string (use Firehose.get_string)
- Simplify get_string_array to use Firehose.get_array
- Simplify get_nested_string using library helpers
- Update decode_record to use Firehose.get_string throughout
The custom WebSocket implementation is kept in the demo as it serves
as the effect handler for the library's WebSocket effects, and is
needed due to server compatibility requirements (ALPN negotiation).
Add three major features to the firehose demo:
1. Event filtering (--filter): Filter by record types (posts, likes, follows,
reposts, blocks, lists, profiles, feeds) or event types (commits,
identities, accounts, handles, tombstones). Supports comma-separated
multiple filters.
2. JSON output (--json): Output events as JSON lines (JSONL) format for
piping to jq or other JSON processing tools. Suppresses human-readable
headers and statistics in this mode.
3. Content extraction (--content): Decode and display actual record content
from CAR blocks embedded in commit events:
- Posts: text, language tags, reply indicators
- Likes/Reposts: subject URI
- Follows/Blocks: target DID
- Profiles: display name and description
The --json and --content flags can be combined for structured output with
full record data.
Example usage:
--filter posts --limit 10 # Show only posts
--json --filter likes # Likes as JSON
--content --filter posts # Posts with text content
--json --content --filter posts # Full JSON with record data
- Show sample firehose events with colored output
- Demonstrate frame encoding/decoding roundtrip
- Add statistics tracking example
- Include integration pattern for WebSocket handlers
- Remove non-functional WebSocket code (requires TLS setup)
- Add Float variant to DAG-CBOR value type with decode_mode (Ipld/Atproto)
- Floats encoded as 64-bit IEEE 754 doubles, NaN/Infinity rejected
- Add CIDv0 support: parsing, encoding, version detection
- Add CBOR strictness checking: reject indefinite-length encoding
- Implement DAG-JSON codec with {/: cid} links and {/: {bytes: b64}} bytes
- Add 22 new IPLD tests (66 total), all 494 compliance tests pass
- Add README.md with project overview and package descriptions
- Add CONTRIBUTING.md with development guidelines
- Add doc/index.mld with odoc documentation entry point
- Add odoc dependency to dune-project for documentation generation
- Enhance module-level docs for effects, ipld, lexicon, mst, and repo packages
Compliance Report Generator:
- Add test/compliance/ with compliance_report.ml and run_compliance.ml
- Generate JSON, Markdown, and HTML reports from atproto-interop-tests
- Achieve 100% pass rate (494/494 tests) across all test suites:
- Syntax Validation: 448/448 (handle, DID, NSID, TID, record key, AT-URI, datetime, language)
- Cryptography: 12/12 (signature verification, P-256/K-256 did:key)
- Data Model (IPLD): 21/21 (DAG-CBOR/CID computation, CID syntax)
- Merkle Search Tree: 13/13 (key heights, common prefix)
Generated reports:
- compliance-report.json (machine-readable)
- COMPLIANCE.md (human-readable Markdown)
- compliance-report.html (interactive HTML with styling)
All core functionality implemented:
- 11 packages (syntax, crypto, multibase, ipld, mst, repo, identity, xrpc, sync, lexicon, api)
- 272 passing tests covering all 42 interop fixtures
- Effects-based I/O for runtime-agnostic code
- Firehose demo example
Documentation task remains open for future work.
Shows how to use the firehose module with OCaml 5 effects for
subscribing to AT Protocol real-time events (commits, identity
changes, etc.).
Implement comprehensive AT Protocol support in OCaml with 11 packages:
- atproto-syntax: Identifier parsing (handle, DID, NSID, TID, AT-URI, etc.)
- atproto-crypto: P-256/K-256 cryptography, did:key, JWT
- atproto-multibase: Base32, Base58btc encoding
- atproto-ipld: DAG-CBOR, CIDs, CAR files, blobs
- atproto-mst: Merkle Search Tree implementation
- atproto-repo: Repository operations and commits
- atproto-identity: DID and handle resolution
- atproto-xrpc: HTTP API client/server, OAuth
- atproto-sync: Firehose and repository synchronization
- atproto-lexicon: Schema language parser, validator, codegen
- atproto-api: High-level client API with rich text
All 272 tests pass, covering all 42 fixture files from atproto-interop-tests.
Uses OCaml 5.4 effects for I/O abstraction.
- Bump version to 0.1.2
- Remove examples from atproto package (fix missing cohttp-eio error)
- Migrate all examples from cohttp-eio to hcs library:
- bsky_bot: use Hcs.Client for HTTP requests
- identity_tool: use Hcs.Client for HTTP requests
- repo_inspector: use Hcs.Client with 100MB response limit
- feed_generator: use Hcs.Websocket for firehose connection
- Update dune files to use hcs instead of cohttp-eio/tls-eio
- All 311 tests pass
- Remove all yojson dependencies from dune-project (fully migrated to atproto-json)
- Add atproto-json dependency to packages that need JSON (lexicon, identity, api)
- mirage-crypto-ec and mirage-crypto-rng already require >= 2.0.2
Note: mirage-crypto-rng-eio 1.2.0 pins all mirage-crypto packages to 1.2.0
which has a P-256 signature verification bug. Do not install it.
- Replace Yojson with Atproto_json (simdjson-backed) in all test files:
- test/mst/test_mst.ml
- test/crypto/test_crypto.ml
- test/sync/test_sync.ml
- test/ipld/test_ipld.ml
- test/compliance/compliance_report.ml
- test/compliance/run_compliance.ml
- Remove yojson dependency from all test dune files
- Add encode_pretty function to lib/json/atproto_json.ml for formatted JSON output
- Update test/lexicon/dune to remove stale yojson dependency
All tests pass except pre-existing crypto signature verification failures.
- Add Atproto_json wrapper around Simdjsont with helpers
- Migrate lib/xrpc/{client,oauth,server}.ml
- Migrate lib/lexicon/{parser,validator,codegen}.ml
- Migrate lib/crypto/jwt.ml
- Migrate lib/ipld/{dag_json,blob}.ml
- Migrate lib/identity/did_resolver.ml
- Migrate lib/sync/firehose.ml
- Migrate lib/api/{agent,richtext}.ml
- Download 315 official AT Protocol lexicon schemas
- Update dune files to use atproto-json package
All lib/ code now uses Atproto_json exclusively.
No direct Simdjsont or Yojson usage in lib/ (except Atproto_json impl).
Tests use Yojson only for fixture loading - not production code.
Add four example tools demonstrating AT Protocol library usage:
- bsky_bot: Bluesky bot for posting, timeline, profiles, following
- feed_generator: Firehose client with filtering and feed skeleton
(evolved from firehose_demo with keyword filtering)
- identity_tool: Handle/DID resolution and identity verification
- repo_inspector: Repository exploration with auto PDS resolution
Key improvements:
- repo_inspector now auto-resolves PDS from DID document
- Added --pds option to skip auto-resolution
- Fixed URL encoding issue (? was incorrectly encoded as %3F)
Add GUIDE.md showing complete workflow: firehose -> identity -> repo
Remove deprecated firehose_demo in favor of feed_generator.
The Lwt_main.run call blocks and doesn't respond to signals by itself.
This fix uses Lwt.pick to race the WebSocket recv against an interrupt
waiter that gets woken when SIGINT is received.
- Add trigger_interrupt() that wakes up the interrupt waiter
- recv() now races recv_impl() against the interrupt signal
- Signal handler calls trigger_interrupt() instead of just setting flag
Now pressing Ctrl+C properly stops the demo and shows statistics.
The websocket-lwt-unix library delivers individual WebSocket frames,
not complete messages. Large messages can be split across multiple
frames (Binary/Text followed by Continuation frames until final=true).
This fix:
- Adds frag_buf buffer to accumulate fragmented message data
- Tracks message type (Binary/Text) via frag_opcode
- Only returns complete binary messages when final frame received
- Properly handles fragmented text messages (skipped)
Without this fix, partial binary data was passed to the CBOR decoder
causing 'invalid payload CBOR' errors after ~1-2 seconds of operation.
- Run indefinitely until Ctrl+C, then print stats and exit cleanly
- Use Lwt_unix.on_signal for proper signal handling integration
- Suppress error messages on graceful exit (decode errors are expected)
- Check interrupted flag before and after WebSocket reads
- Add 'Press Ctrl+C to stop' message at startup
- Use Dag_json.encode_string for record serialization instead of custom code
- Add Climate for CLI parsing with auto-generated help
- Add compact stats with SIGINT handling (prints on Ctrl+C)
- Remove custom record_content type and JSON escaping
- Simplify filter names to single form only
Line count: 805 → 547 (32% reduction)
Major architectural change:
- Use Firehose.subscribe from the library instead of manual event loop
- Move WebSocket implementation into Ws_handler module as effect handler
- Demo now focuses on filtering/formatting, not protocol details
The demo provides effect handlers for:
- Firehose.Ws_connect: TLS connection with WebSocket upgrade
- Firehose.Ws_recv: Frame parsing with fragmentation support
- Firehose.Ws_close: Connection cleanup
This separates concerns:
- Library: protocol logic (subscribe, decode_frame, event types)
- Demo: presentation (filtering, formatting, statistics)
- Ws_handler: I/O implementation (TLS, WebSocket framing)
Line count: 795 (down from 1204, -34%)
Major changes:
- Remove demo mode and sample data (~143 lines)
- Consolidate JSON formatting with shared helpers (json_opt, json_str)
- Merge duplicate formatting functions (format_event, format_event_json now take ~rich param)
- Simplify filter matching with prefix_match helper
- Compact statistics printing
- Remove verbose section comment banners
- Use more concise OCaml idioms throughout
All features preserved:
- --filter for event type filtering
- --json for JSON lines output
- --content for record content extraction
- --cursor for resuming from sequence
- --limit for stopping after N events
Remove duplicate CBOR helper functions and use Firehose module's
get_string, get_array, etc. instead of local reimplementations.
Changes:
- Remove get_cbor_string (use Firehose.get_string)
- Simplify get_string_array to use Firehose.get_array
- Simplify get_nested_string using library helpers
- Update decode_record to use Firehose.get_string throughout
The custom WebSocket implementation is kept in the demo as it serves
as the effect handler for the library's WebSocket effects, and is
needed due to server compatibility requirements (ALPN negotiation).
Add three major features to the firehose demo:
1. Event filtering (--filter): Filter by record types (posts, likes, follows,
reposts, blocks, lists, profiles, feeds) or event types (commits,
identities, accounts, handles, tombstones). Supports comma-separated
multiple filters.
2. JSON output (--json): Output events as JSON lines (JSONL) format for
piping to jq or other JSON processing tools. Suppresses human-readable
headers and statistics in this mode.
3. Content extraction (--content): Decode and display actual record content
from CAR blocks embedded in commit events:
- Posts: text, language tags, reply indicators
- Likes/Reposts: subject URI
- Follows/Blocks: target DID
- Profiles: display name and description
The --json and --content flags can be combined for structured output with
full record data.
Example usage:
--filter posts --limit 10 # Show only posts
--json --filter likes # Likes as JSON
--content --filter posts # Posts with text content
--json --content --filter posts # Full JSON with record data
- Add Float variant to DAG-CBOR value type with decode_mode (Ipld/Atproto)
- Floats encoded as 64-bit IEEE 754 doubles, NaN/Infinity rejected
- Add CIDv0 support: parsing, encoding, version detection
- Add CBOR strictness checking: reject indefinite-length encoding
- Implement DAG-JSON codec with {/: cid} links and {/: {bytes: b64}} bytes
- Add 22 new IPLD tests (66 total), all 494 compliance tests pass
- Add README.md with project overview and package descriptions
- Add CONTRIBUTING.md with development guidelines
- Add doc/index.mld with odoc documentation entry point
- Add odoc dependency to dune-project for documentation generation
- Enhance module-level docs for effects, ipld, lexicon, mst, and repo packages
Compliance Report Generator:
- Add test/compliance/ with compliance_report.ml and run_compliance.ml
- Generate JSON, Markdown, and HTML reports from atproto-interop-tests
- Achieve 100% pass rate (494/494 tests) across all test suites:
- Syntax Validation: 448/448 (handle, DID, NSID, TID, record key, AT-URI, datetime, language)
- Cryptography: 12/12 (signature verification, P-256/K-256 did:key)
- Data Model (IPLD): 21/21 (DAG-CBOR/CID computation, CID syntax)
- Merkle Search Tree: 13/13 (key heights, common prefix)
Generated reports:
- compliance-report.json (machine-readable)
- COMPLIANCE.md (human-readable Markdown)
- compliance-report.html (interactive HTML with styling)
Implement comprehensive AT Protocol support in OCaml with 11 packages:
- atproto-syntax: Identifier parsing (handle, DID, NSID, TID, AT-URI, etc.)
- atproto-crypto: P-256/K-256 cryptography, did:key, JWT
- atproto-multibase: Base32, Base58btc encoding
- atproto-ipld: DAG-CBOR, CIDs, CAR files, blobs
- atproto-mst: Merkle Search Tree implementation
- atproto-repo: Repository operations and commits
- atproto-identity: DID and handle resolution
- atproto-xrpc: HTTP API client/server, OAuth
- atproto-sync: Firehose and repository synchronization
- atproto-lexicon: Schema language parser, validator, codegen
- atproto-api: High-level client API with rich text
All 272 tests pass, covering all 42 fixture files from atproto-interop-tests.
Uses OCaml 5.4 effects for I/O abstraction.