notes
about things notes.zzstoatzz.io
notes
Star
Fork
Watch
notes languages ziglang std compression.md
2.5 kB · 51 lines
Markdown
at main

compression #

checked against zig 0.16.0 and 0.17.0.

std.compress.zstd (decode) #

zig std can decompress zstandard — no vendored libzstd needed for plain-frame decode:

var in: std.Io.Reader = .fixed(compressed);
var d: std.compress.zstd.Decompress = .init(&in, &.{}, .{});
_ = try d.reader.streamRemaining(&allocating_writer.writer);

measure before shipping it on a hot path: on real jss archive blocks (atproto-bench jss lane, 58 frames / 61 MB decompressed) std's decoder took ~410 ms where libzstd took 22 ms — ~20× — and the gap survived every buffering strategy (allocating writer, reused window + discard, preallocated). vendoring facebook/zstd decode-only is about 10 C files and a ~150-line build.zig addition, and it cross-compiles. std zstd is fine when the wire is the bottleneck.

sharp edges, the same in the 0.16.0 and 0.17.0 source:

  • decode only. there is no compressor in std.
  • verify_checksum: true panics — "not implemented yet" in Decompress.zig. frame content CRCs are silently skipped, so integrity must come from an outer layer (e.g. a per-segment xxh3 checksum; std.hash.XxHash3 has streaming init/update/final).
  • no dictionary support: a frame with a dictionary id fails with DictionaryIdFlagUnsupported. fine for plain frames; dictionary-compressed streams (jetstream's subscribe-v2 wire compression) need libzstd.
  • with a non-empty buffer, init asserts capacity ≥ options.window_len + zstd.block_size_max; the empty-buffer + writer form (above) is the simple path for buffered whole-frame decode.

dictionary decoding with libzstd #

with vendored libzstd, digest the dictionary once (ZSTD_createDDict) and reuse it and one ZSTD_DCtx across frames (ZSTD_decompress_usingDDict); building either per message puts setup work on the hot path. bound the decompressed size separately from the wire size. measure decode CPU and bytes on the wire separately: a dictionary that saves bytes can cost CPU, and fetching it is a startup cost.

sources #

  • zig 0.16.0 and 0.17.0 lib/std/compress/zstd/Decompress.zig (verify_checksum, DictionaryIdFlagUnsupported)
  • zat: vendored zstd v1.5.7 decode-only, jss archive block decode
  • jetstream src/zstd.zig (DDict bindings); atproto-bench jetstream-compression/README.md, 2026-09-06