binary encoding #

written against zig 0.15; the writer code is checked against 0.17.0.

encoding and decoding binary wire formats (CBOR, CAR, protocol frames): raw bytes, endianness, varints, and content addressing.

encoders take *std.Io.Writer #

since 0.15 an encoder takes the concrete *std.Io.Writer, and the caller decides where the bytes go: a fixed buffer, a growing allocation, a file, a socket.

pub fn encode(allocator: Allocator, w: *std.Io.Writer, value: Value) !void {
    switch (value) {
        .unsigned => |v| try writeArgument(w, 0, v),
        .text => |t| {
            try writeArgument(w, 3, t.len);
            try w.writeAll(t);
        },
        .map => |entries| {
            // DAG-CBOR sorts keys, which needs scratch memory
            const sorted = try allocator.dupe(MapEntry, entries);
            defer allocator.free(sorted);
            std.mem.sort(MapEntry, sorted, {}, dagCborKeyLessThan);
            // ...
        },
        // ...
    }
}

the allocator is for temporary work during encoding (sorting map keys, intermediate buffers), not for the output.

// fixed buffer, no allocation for the output
var buf: [1024]u8 = undefined;
var w: std.Io.Writer = .fixed(&buf);
try encode(alloc, &w, value);
const bytes = w.buffered();

// growing buffer, caller owns the result
pub fn encodeAlloc(allocator: Allocator, value: Value) ![]u8 {
    var aw: std.Io.Writer.Allocating = .init(allocator);
    errdefer aw.deinit();
    try encode(allocator, &aw.writer, value);
    return aw.toOwnedSlice();
}

0.15 code written against anytype writers (std.io.fixedBufferStream, ArrayList(u8).writer(gpa)) has to move to this form: both are gone in 0.16.

big-endian integers and LEB128 #

std.Io.Writer has writeInt(T, value, .big), and a CBOR argument is a prefix byte plus a big-endian integer:

fn writeArgument(w: *std.Io.Writer, major: u3, val: u64) !void {
    const prefix: u8 = @as(u8, major) << 5;
    if (val <= 0xffff) {
        try w.writeByte(prefix | 25);
        try w.writeInt(u16, @intCast(val), .big);
    }
    // ...
}

unsigned varints (LEB128, used by CID, CAR and other IPLD formats) are w.writeUleb128(val) on the writer and r.takeLeb128(u64) on a std.Io.Reader, which also rejects a varint too long for the result type:

var r: std.Io.Reader = .fixed(data);
const len = try r.takeLeb128(u64);
const rest = data[r.seek..];

arena per message #

for streaming protocols, create an arena per incoming message. all decoding allocations go into it, then free everything at once:

pub fn serverMessage(self: *Self, data: []const u8) !void {
    var arena = std.heap.ArenaAllocator.init(self.allocator);
    defer arena.deinit();

    const event = decodeFrame(arena.allocator(), data) catch |err| {
        log.debug("decode error: {s}", .{@errorName(err)});
        return;
    };

    self.handler.onEvent(event);
    // arena freed here — all decoded data is gone
}

this means the handler's onEvent must not hold references to event data past the call. if it needs to, it must copy into its own allocator.

specialized decoders #

when generic decoding is too expensive, write a purpose-built parser for a known schema. the generic path builds Value unions, MapEntry arrays, and handles every CBOR type. if you know the exact shape, skip all that.

example: MST nodes are always map(2) { "e": array[entries...], "l": CID|null }. instead of cbor.decodeAll() → extract fields from Value unions, parse the CBOR bytes directly:

pub fn decodeMstNode(allocator: Allocator, data: []const u8) MstDecodeError!MstNodeData {
    // expect map(2), key "e", array(n) — known byte sequence
    // parse entries inline, zero-copy slicing into input buffer
    // only allocation: the entries array itself
}

pub const MstNodeData = struct {
    left: ?[]const u8,       // raw CID bytes (borrowed from input)
    entries: []MstEntryData, // heap-allocated array
};

pub const MstEntryData = struct {
    prefix_len: usize,
    key_suffix: []const u8,  // borrowed from input
    value_cid: []const u8,   // borrowed from input
    tree: ?[]const u8,       // borrowed from input
};

on an MST walk the specialized decoder is a modest gain over the generic one; the larger gain is verifying structure during the walk instead of rebuilding the whole tree first.

when to use this pattern:

• you decode the same schema thousands of times (MST nodes, CBOR blocks)
• the schema is stable and well-known
• profiling shows decode time dominates

when NOT to use it:

• the schema varies or is user-defined
• you only decode a handful of times
• generic decode is fast enough

deterministic encoding #

DAG-CBOR requires deterministic output (same value → same bytes). the main rules:

• shortest integer encoding: 0-23 inline, 24-255 in 1 byte, etc.
• map keys sorted: by byte length first, then lexicographically
• no floats, no indefinite lengths

sorting map keys during encoding:

fn dagCborKeyLessThan(_: void, a: MapEntry, b: MapEntry) bool {
    if (a.key.len != b.key.len) return a.key.len < b.key.len;
    return std.mem.order(u8, a.key, b.key) == .lt;
}

// in encoder:
const sorted = try allocator.dupe(MapEntry, entries);
defer allocator.free(sorted);
std.mem.sort(MapEntry, sorted, {}, dagCborKeyLessThan);

the dupe + sort pattern avoids mutating the input — the caller's entries slice stays unchanged.

sources #

• zat src/internal/repo/cbor.zig, streaming/firehose.zig, repo/mst.zig: CBOR encoder, per-message arenas, the MST node decoder
• zig 0.17.0 lib/std/Io/Writer.zig (fixed, writeInt, writeUleb128, Allocating), lib/std/Io/Reader.zig (takeLeb128)