---

title: stream #

available in builds compiled with the indexer_stream feature. /stream is hydrant's ordered websocket event stream. each subscriber receives a full copy of the stream.

/stream is not wire- or protocol-compatible with tap. tap uses a sharded work queue, server-managed acknowledgements, and redelivery; hydrant uses broadcast delivery and client-managed cursors.

GET /stream #

subscribe to the ordered event stream.

query parameters #

param	type	description
cursor	integer	inclusive u64 event ID from which to replay durable record events.
stream	string	the stream cursor came from, see /stream/head. when it isn't this one, hydrant sends a StreamMismatch error frame and closes the connection.

if cursor is omitted, the connection starts at the current stream head and receives only new events. if it is present, hydrant replays durable record events whose IDs are greater than or equal to the cursor, up to the head observed when the connection starts, then switches to live delivery.

cursor=0 replays all retained record events. IDs are monotonic but may have gaps: live-only identity and account events consume IDs without being persisted, and events that cannot be inflated are skipped.

event envelope #

events are JSON text frames with this envelope:

{
  "id": 42,
  "type": "record",
  "record": {}
}

field	type	description
id	integer	u64 stream event ID.
type	string	record, identity, or account.
record	object	present when type is record.
identity	object	present when type is identity.
account	object	present when type is account.

only the payload matching type is present.

record #

record events are durable and available through cursor replay.

{
  "id": 42,
  "type": "record",
  "record": {
    "live": true,
    "did": "did:plc:abc123xyz",
    "rev": "3kpjxabc123",
    "collection": "app.bsky.feed.post",
    "rkey": "3kpjxabc123",
    "action": "create",
    "record": {
      "text": "hello, world!",
      "$type": "app.bsky.feed.post",
      "createdAt": "2026-05-27T12:00:00.000Z"
    },
    "cid": "bafyreihy..."
  }
}

field	type	description
live	boolean	true for live ingestion; false for backfill.
did	string	repository DID.
rev	string	repository revision TID.
collection	string	collection NSID.
rkey	string	record key.
action	string	create, update, or delete.
record	object	decoded DAG-CBOR record, its keys in the order the record's CBOR has them (shorter first), not sorted; omitted when no record content is stored.
cid	string	CID of record; omitted when no record content is stored.

record and cid are omitted for deletes and when hydrant is configured not to store record content, including HYDRANT_ONLY_INDEX_LINKS=true.

a redaction stops a body from being served once it completes, except in events already queued for a reader at that moment.

identity #

identity events are live-only and are not available through cursor replay.

{
  "id": 43,
  "type": "identity",
  "identity": {
    "did": "did:plc:abc123xyz",
    "handle": "user.bsky.social"
  }
}

field	type	description
did	string	repository DID.
handle	string	current handle; omitted when unavailable.

account #

account events are live-only and are not available through cursor replay.

{
  "id": 44,
  "type": "account",
  "account": {
    "did": "did:plc:abc123xyz",
    "active": false,
    "status": "deactivated"
  }
}

field	type	description
did	string	repository DID.
active	boolean	whether the repository is active.
status	string	account status; omitted when unavailable.

websocket behavior #

• the server sends an empty ping every 30 seconds and responds to client ping frames with pong frames.
• client text and binary frames are not accepted; hydrant closes the connection when it receives one.
• when /stream is in HYDRANT_THROTTLED_STREAMS (it isn't by default), a replay from a cursor runs at most HYDRANT_STREAM_REPLAY_SPEED times the bandwidth the live stream would take (10 by default), or HYDRANT_STREAM_REPLAY_FLOOR_BYTES a second (about 80 Mbit/s) when that's more. the live tail never waits on it.
• a subscriber that falls behind the live broadcast catches up from the database at its own pace. record events are never skipped, but identity and account events it fell behind on are not replayed.
• if hydrant can't read the next events from the database, it closes the connection rather than skip them. reconnect with the last id you received as the cursor.
• when hydrant stops, it closes the connection with close code 1012 (service restart). reconnect the same way. under systemd socket activation, the reconnect waits for the next hydrant instead of being refused.
• if sending blocks for HYDRANT_STREAM_SEND_TIMEOUT (30 seconds by default), hydrant sends an error text frame and closes the connection:

{
  "type": "error",
  "error": "ConsumerTooSlow",
  "message": "stream socket send blocked for at least 30 seconds"
}

GET /stream/head #

get the ID of the newest durable record event, and the stream it belongs to.

{ "id": 289455, "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71" }

id is null while no events are stored. stream is the id of this database's stream, new for every database, so two nodes, or one that was wiped, never share one. an event id only means something on the stream it came from. identity and account events consume IDs without being persisted, so this is the newest ID a cursor replay can deliver, not the ID counter.

a consumer can snapshot the head before subscribing with cursor=0 and treat the replay as caught up once it has seen a frame with id greater than or equal to the snapshot. keep an idle-timeout fallback: if the head event itself cannot be inflated, it is skipped and never delivered.

chunks #

with HYDRANT_STREAM_CHUNK_SPAN set, hydrant also seals the stream into immutable files a subscriber starting from nothing can read instead of replaying /stream from 0: they're compressed, they cost hydrant a file read instead of rendering every event again for every subscriber, and a cache can hold them.

a chunk covers the ids first through last. like a replay, it holds only record events, never identity or account ones, and every one of its range. reading the chunks in order, then /stream from the id after the last one, delivers exactly what /stream?cursor=0 does.

chunk format #

a chunk is a STAR (DASL's streaming archive, from its pull 54), compressed with zstd. decompressed, it is

2a 6c 01 00 | varint len | meta | ( varint 8 | id | varint len | event )*

• the four magic bytes, then the header: an unsigned varint (LEB128, no superfluous trailing 00 byte, at most 9 bytes) holding the length of meta, a DRISL map:

  key	value
  $type	"systems.gaze.hydrant.streamChunk"
  stream	the stream id, as in /stream/head
  first	the first id the chunk covers
  last	the last id the chunk covers

• then one pair per event, in id order, each id between first and last and greater than the one before. the key is 8 bytes, the id big-endian. the value is a DRISL map of the event's /stream frame fields, with the record as hydrant stores it:

  key	DRISL	json frame
  live	bool	record.live
  did	text	record.did
  rev	text	record.rev
  collection	text	record.collection
  rkey	text	record.rkey
  action	text	record.action
  record	byte string, absent when there's no body	record.record
  cid	tag 42 cid link, absent for a delete and when it's the cidv1 (dag-cbor, sha2-256) of record's bytes	record.cid, as a base32 string

a missing cid next to a record is the cidv1 of the bytes as they are, not of a re-encode of them. the archive ends after the last pair, with nothing after it. STAR can't tell an archive cut short at a pair boundary from a whole one, so check a chunk against its sha256 before trusting it.

the compressed file is several zstd frames back to back, which decompress as one stream: one for the magic and header, then one or more holding the pairs. that is how a merge works: a new header frame, then each chunk's frames after its own header, as they are.

record holds the record's dag-cbor bytes. the json frame has them as json: maps keep their cbor key order, a tag 42 link becomes {"$link": "<base32 cid>"}, bytes become {"$bytes": "<padded base64>"}, ints, strings, bools, nulls and arrays map as themselves. a body this can't express faithfully (a float, an int past 64 bits, a map key $link or $bytes, keys out of dag-cbor order or repeated, a blob map with more or other fields than $type, ref, mimeType and size) is written the way jacquard's RawData serializes it, and one that isn't dag-cbor at all leaves record out of the json frame. the json frame is then

{"id":<id>,"type":"record","record":{"live":..,"did":..,"rev":..,"collection":..,"rkey":..,"action":..[,"record":<body>][,"cid":"<cid>"]}}

with no whitespace, which is byte for byte what /stream sends.

new chunks are one span each. hydrant merges the newest 4 chunks of the same size into one, up to 16 spans, so the list grows by one chunk every 16 spans and a subscriber can fetch several at once. it builds them on a low priority thread, and while ingest is behind it seals one span every 30 seconds, so catching up on a whole stream (turning chunks on, or after a wipe) waits for ingest.

a redaction or an erase reaches every chunk holding the repository's events. those chunks leave the list until hydrant has built them again from the event rows, with the bodies gone like a replay would have them. the stream itself has no redaction event, so a subscriber that already read a chunk, like one already connected to /stream, still holds what it read.

GET /stream/manifest #

the chunks to read, in order.

{
  "stream": "5b0e6c8f2a1d4e7b9c3f0a6d8e2b4c71",
  "generation": 41,
  "through": 917503,
  "chunks": [
    { "first": 786432, "last": 851967, "events": 47512, "bytes": 5104330, "sha256": "9f2c..." },
    { "first": 851968, "last": 917503, "events": 61664, "bytes": 11420553, "sha256": "41ab..." }
  ]
}

(the list is shortened here.) the chunks run from id 0 through through with no holes, which can end before the newest chunk while one is being rebuilt. read them, then /stream?cursor={through + 1}&stream={stream}. if a chunk can't be read, the frames read so far are still a valid place to go on from: continue with /stream from the id after the last frame read. generation changes whenever the list does. 404 when chunks are off.

GET /stream/chunk/{sha256} #

the chunk's bytes as stored, application/zstd, whose sha256 is the name and the ETag. a chunk never changes, so it can be cached for good, and a Range request gets part of it. a merge leaves the chunks it replaced readable for 15 minutes, for subscribers part way through an older list. a chunk a redaction or erase reached is 404 at once.