vibescrobble! every agent, live on stage. vibescrobble.com

fix(index): bound websocket messages and decoded lengths from upstream streams #809

targeting main from claude/ingest-bounds
Merged opened by and assigned to quaternion-karabela.test.pds.did.bot
targeting main from claude/ingest-bounds

The index now sets explicit websocket limits instead of tungstenite's defaults (64 MiB per message, 16 MiB per frame): 4 MiB for both Jetstream and the relay or PDS firehose. Both are sized so legitimate traffic cannot reach them. A record can be about 1 MB and grows when rendered as JSON. The atproto spec caps a #commit at 2,000,000 bytes of blocks, and a #sync carries only the signed commit block. A message over a limit is treated as a gap, not retried. The client logs one warning naming the stream and size, drops its cursor, reconnects at the live head, and reports the existing Update::Gap, so the usual catch-up runs. Retrying the same cursor would have been worse: the stream would send the same oversized event again, forever. One oversized record could stall ingest for good; now it costs a gap the length of a reconnect, plus the catch-up that a gap already triggers.

The CBOR and CAR decoders now bound length claims by the bytes that arrived. DAG-CBOR arrays and maps are capped at 64Ki elements, with allocation limited to the frame's size. A CAR block may announce at most 1 MiB, the spec's record limit plus its CID, and must fit in what remains of the frame before anything is allocated. Before, a few bytes could ask for a huge allocation. The rest of ingest already had bounds, all checked: HTTP bodies at 4 MiB, DID documents at 64 KiB, profiles and avatars, and record pages.

tests/stream_bounds.rs runs fake Jetstream and firehose servers that replay a normal message and then an oversized one every time they are asked for the same cursor. The client asks once with the cursor, then without it, reports the gap, and receives the messages that follow. frames.rs tests refuse an oversized CBOR array claim and a CAR block longer than its frame. The index tests pass.

🤖 Generated with Claude Code

Commits 3

fix(index): treat an oversized stream message as a gap, not a retry
fix(index): bound DAG-CBOR and CAR length claims
fix(index): bound websocket messages from the relay and Jetstream