# binary encoding

> written against zig 0.15; the writer code is checked against 0.17.0.

encoding and decoding binary wire formats (CBOR, CAR, protocol frames): raw
bytes, endianness, varints, and content addressing.

## encoders take `*std.Io.Writer`

since 0.15 an encoder takes the concrete `*std.Io.Writer`, and the caller
decides where the bytes go: a fixed buffer, a growing allocation, a file, a
socket.

```zig
pub fn encode(allocator: Allocator, w: *std.Io.Writer, value: Value) !void {
    switch (value) {
        .unsigned => |v| try writeArgument(w, 0, v),
        .text => |t| {
            try writeArgument(w, 3, t.len);
            try w.writeAll(t);
        },
        .map => |entries| {
            // DAG-CBOR sorts keys, which needs scratch memory
            const sorted = try allocator.dupe(MapEntry, entries);
            defer allocator.free(sorted);
            std.mem.sort(MapEntry, sorted, {}, dagCborKeyLessThan);
            // ...
        },
        // ...
    }
}
```

the allocator is for temporary work during encoding (sorting map keys,
intermediate buffers), not for the output.

```zig
// fixed buffer, no allocation for the output
var buf: [1024]u8 = undefined;
var w: std.Io.Writer = .fixed(&buf);
try encode(alloc, &w, value);
const bytes = w.buffered();

// growing buffer, caller owns the result
pub fn encodeAlloc(allocator: Allocator, value: Value) ![]u8 {
    var aw: std.Io.Writer.Allocating = .init(allocator);
    errdefer aw.deinit();
    try encode(allocator, &aw.writer, value);
    return aw.toOwnedSlice();
}
```

0.15 code written against `anytype` writers (`std.io.fixedBufferStream`,
`ArrayList(u8).writer(gpa)`) has to move to this form: both are gone in 0.16.

## big-endian integers and LEB128

`std.Io.Writer` has `writeInt(T, value, .big)`, and a CBOR argument is a
prefix byte plus a big-endian integer:

```zig
fn writeArgument(w: *std.Io.Writer, major: u3, val: u64) !void {
    const prefix: u8 = @as(u8, major) << 5;
    if (val <= 0xffff) {
        try w.writeByte(prefix | 25);
        try w.writeInt(u16, @intCast(val), .big);
    }
    // ...
}
```

unsigned varints (LEB128, used by CID, CAR and other IPLD formats) are
`w.writeUleb128(val)` on the writer and `r.takeLeb128(u64)` on a
`std.Io.Reader`, which also rejects a varint too long for the result type:

```zig
var r: std.Io.Reader = .fixed(data);
const len = try r.takeLeb128(u64);
const rest = data[r.seek..];
```

## arena per message

for streaming protocols, create an arena per incoming message. all decoding allocations go into it, then free everything at once:

```zig
pub fn serverMessage(self: *Self, data: []const u8) !void {
    var arena = std.heap.ArenaAllocator.init(self.allocator);
    defer arena.deinit();

    const event = decodeFrame(arena.allocator(), data) catch |err| {
        log.debug("decode error: {s}", .{@errorName(err)});
        return;
    };

    self.handler.onEvent(event);
    // arena freed here — all decoded data is gone
}
```

this means the handler's `onEvent` must not hold references to event data past the call. if it needs to, it must copy into its own allocator.

## specialized decoders

when generic decoding is too expensive, write a purpose-built parser for a known schema. the generic path builds `Value` unions, `MapEntry` arrays, and handles every CBOR type. if you know the exact shape, skip all that.

example: MST nodes are always `map(2) { "e": array[entries...], "l": CID|null }`. instead of `cbor.decodeAll()` → extract fields from Value unions, parse the CBOR bytes directly:

```zig
pub fn decodeMstNode(allocator: Allocator, data: []const u8) MstDecodeError!MstNodeData {
    // expect map(2), key "e", array(n) — known byte sequence
    // parse entries inline, zero-copy slicing into input buffer
    // only allocation: the entries array itself
}

pub const MstNodeData = struct {
    left: ?[]const u8,       // raw CID bytes (borrowed from input)
    entries: []MstEntryData, // heap-allocated array
};

pub const MstEntryData = struct {
    prefix_len: usize,
    key_suffix: []const u8,  // borrowed from input
    value_cid: []const u8,   // borrowed from input
    tree: ?[]const u8,       // borrowed from input
};
```

on an MST walk the specialized decoder is a modest gain over the generic one; the larger gain is verifying structure during the walk instead of rebuilding the whole tree first.

when to use this pattern:
- you decode the same schema thousands of times (MST nodes, CBOR blocks)
- the schema is stable and well-known
- profiling shows decode time dominates

when NOT to use it:
- the schema varies or is user-defined
- you only decode a handful of times
- generic decode is fast enough

## deterministic encoding

DAG-CBOR requires deterministic output (same value → same bytes). the main rules:

- **shortest integer encoding**: 0-23 inline, 24-255 in 1 byte, etc.
- **map keys sorted**: by byte length first, then lexicographically
- **no floats, no indefinite lengths**

sorting map keys during encoding:

```zig
fn dagCborKeyLessThan(_: void, a: MapEntry, b: MapEntry) bool {
    if (a.key.len != b.key.len) return a.key.len < b.key.len;
    return std.mem.order(u8, a.key, b.key) == .lt;
}

// in encoder:
const sorted = try allocator.dupe(MapEntry, entries);
defer allocator.free(sorted);
std.mem.sort(MapEntry, sorted, {}, dagCborKeyLessThan);
```

the dupe + sort pattern avoids mutating the input — the caller's `entries` slice stays unchanged.

## sources

- [zat](https://tangled.org/zat.dev/zat) `src/internal/repo/cbor.zig`, `streaming/firehose.zig`, `repo/mst.zig`: CBOR encoder, per-message arenas, the MST node decoder
- zig 0.17.0 `lib/std/Io/Writer.zig` (`fixed`, `writeInt`, `writeUleb128`, `Allocating`), `lib/std/Io/Reader.zig` (`takeLeb128`)