zig-ogg #

A reader and writer for the Ogg container format, written in Zig against RFC 3533, the specification of the format. It is a library first: it has no filesystem, no threads and no dependencies, reads from a std.Io.Reader and writes to a std.Io.Writer, and leaves the question of where the bytes come from to whoever is calling it. A command-line tool comes with it, mostly to prove the library can be used.

$ zig-ogg info song.ogg
song.ogg: 412 pages, 1 logical bitstream
  serial 0x784d00be  Vorbis audio       2104 packets, 3015291 bytes, ends at granule 2646000
$ zig-ogg check song.ogg
song.ogg: OK, 412 pages, 2104 packets

The API documentation is generated from the doc comments, where most of the explanation lives, and is published at https://jeff.jcollie.page/zig-ogg/.

What Ogg is, and what it is not #

Ogg is framing and nothing else. It takes packets from a codec, chops them into pages of a few kilobytes with a checksum and a position marker on each, and interleaves the pages of however many logical bitstreams share the file. It knows nothing about what the packets mean — not even how long a second is, since "Ogg does not have a concept of 'time': it only knows about sequentially increasing, unitless position markers" (RFC 3533, Section 4).

So this library moves packets in and out of pages and leaves every question about their contents to whoever wrote them. It does not decode Vorbis, FLAC, Opus or anything else; what it will tell you is which of them a stream claims to be, because that is written in the first page and is the container's business.

Ogg files are application/ogg, audio/ogg or video/ogg depending on what they carry, per RFC 5334; .ogg, .oga, .ogv and .opus are the extensions in use.

What it does #

Pages are parsed and written in full: the capture pattern, the version, all three header flags, the granule position with its -1 sentinel, the serial and sequence numbers, the checksum, and the segment table with the lacing rules that make a packet's length unbounded. A page that parsed is written back byte for byte.

Demultiplexing handles both kinds of multiplexing RFC 3533 defines, and does not need to be told which it is looking at. "Grouping" interleaves the pages of concurrent logical bitstreams; "chaining" concatenates whole streams end to end; both are just pages keyed by serial number, so a demuxer that looks the number up and reassembles per stream handles either, or the two together.

Recovering from damage is what the capture pattern and the per-page checksum are for, and PageReader.resync uses them: it steps over corrupt bytes until a page whose checksum is right turns up, and says how many it threw away. A stream joined in the middle, a page lost from the sequence, and a page that contradicts the one before it about whether a packet was still open are all reported rather than quietly papered over — the packet that follows arrives with after_hole set.

Writing puts packets into pages, choosing page boundaries by a target size, and gets the parts right that are easy to get wrong: the end-of-stream flag lands on a page with content on it, the beginning-of-stream page carries the identification packet alone, a page where no packet finishes carries no granule position, and a packet whose length is a multiple of 255 gets its terminating zero.

Not implemented: seeking. The granule positions and the page index are all there, and bisecting on them is how seeking in Ogg is done, but nothing here does it yet — it needs a seekable input, which the std.Io.Reader this takes is not required to be. Nor is there a muxer that decides for you how to interleave several streams' pages; writing two streams into one file is done by pointing two StreamWriters at the same output, which leaves the interleaving policy where the caller can see it.

Using the library #

Fetch it and import the module it exposes as ogg:

$ zig fetch --save git+https://git.jcollie.dev/jeff/zig-ogg.git

const ogg = b.dependency("ogg", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("ogg", ogg.module("ogg"));

Reading packets #

Demuxer is the whole of demultiplexing. Packets come out in the order the pages carrying them appear, each tagged with the serial number of the logical bitstream it belongs to:

const ogg = @import("ogg");

var buffer: [ogg.Page.max_size]u8 = undefined;
var d: ogg.Demuxer = .init(gpa, reader, &buffer, .{});
defer d.deinit();

while (try d.next()) |packet| {
    if (packet.first) {
        // The first packet of a stream identifies its codec.
        const codec = ogg.codec.identify(packet.data) orelse continue;
        std.debug.print("0x{x}: {s}\n", .{ packet.serial_number, codec.description() });
    }
    // packet.data is valid until the next call to next().
}

A Packet carries a little more than its bytes:

Field	What it says
serial_number	Which logical bitstream it came from.
data	The packet. Valid until the next call to next.
granule_position	Where the codec's position counter stood afterwards, when the stream said — which is only for the last packet to finish on a page, since a page carries one position and that is whose it is. Null otherwise.
index	Counts packets within the logical bitstream from zero.
first, last	It came off the bos page, or finished on the eos page.
after_hole	Something before this packet was lost. The packet itself is whole.

Writing packets #

One StreamWriter is one logical bitstream:

var body: [ogg.Page.max_body_size]u8 = undefined;
var w: ogg.StreamWriter = .init(writer, &body, .{ .serial_number = serial });

try w.writePacket(identification_header, 0);
try w.writePacket(comment_header, 0);
for (audio[0 .. audio.len - 1]) |frame| try w.writePacket(frame.bytes, frame.samples);
try w.writeLastPacket(audio[audio.len - 1].bytes, total_samples);

writeLastPacket is worth using wherever the last packet is known, so that the end-of-stream flag goes on the page carrying it rather than on an empty page after it. flush ends the current page early, which is what a live stream wants and what the mapping's rule about the bos page needs; finish ends the stream.

Two streams in one file — grouping — is two writers pointed at the same output. The format requires every bos page to come before any other page, which is why the writer does not hold those two back:

var a: ogg.StreamWriter = .init(writer, &body_a, .{ .serial_number = 100 });
var b: ogg.StreamWriter = .init(writer, &body_b, .{ .serial_number = 200 });
try a.writePacket(a_header, 0);   // both bos pages go out at once...
try b.writePacket(b_header, 0);
// ...and only then does content follow, in whatever order suits.

One stream after another — chaining — needs nothing at all: finish one and start the next.

Pages on their own #

Below the demuxer, PageReader hands back one Page at a time for a caller that wants the framing rather than the contents, and Page.parse is the parser by itself, over bytes, with no I/O:

var pr: ogg.PageReader = .init(reader, &buffer);
while (try pr.next()) |page| {
    // page.lacing, page.body, page.packets() ...
}

Page.packets() walks the packet fragments a page holds, saying of each whether it is the tail of one that began earlier and whether it ends here. That is the layer Demuxer is built on, and is what to use to do something else with.

What allocates, and what does not #

A page is bounded and a packet is not, and that one fact decides the shape of everything here.

A segment table holds at most 255 entries of at most 255 bytes, so no page exceeds 65307 bytes however hostile the stream is. Page, PageReader and StreamWriter therefore allocate nothing at all: each works out of one buffer the caller hands it, sized by the Page.max_size and Page.max_body_size constants, and a page read is handed back as slices into that buffer rather than copied out of it.

A packet, on the other hand, can be continued from page to page without end. So Demuxer — the only part that reassembles one — takes an allocator, and takes a max_packet_size to refuse at, because otherwise a stream whose every page ends mid-packet is a request for all the memory there is. max_streams is the same answer to a stream that names a new serial number on every page. Even there, nothing is copied in the ordinary case: a packet that begins and ends inside one page points into the page buffer.

The command-line tool #

$ zig-ogg info <file.ogg>...     # what logical bitstreams a file holds
$ zig-ogg pages <file.ogg>       # one line per page
$ zig-ogg packets <file.ogg>     # one line per packet
$ zig-ogg check <file.ogg>...    # verify every page's checksum and order

pages is the one to reach for when a file is behaving oddly, since it shows the framing decisions another muxer made:

$ zig-ogg pages src/tests/data/grouped.ogg
         0  serial 0x6a32f4ce  page 0      -b-   1 segments,    30 bytes, granule 0
        58  serial 0xcf6f1915  page 0      -b-   1 segments,    19 bytes, granule 0
       105  serial 0x6a32f4ce  page 1      ---  11 segments,  2538 bytes, granule 0
      2681  serial 0xcf6f1915  page 1      ---   1 segments,    60 bytes, granule 0
      2769  serial 0x6a32f4ce  page 2      --e  25 segments,   447 bytes, granule 6000
      3268  serial 0xcf6f1915  page 2      --e  38 segments,  3194 bytes, granule 36312

The three flag columns are continued, bos and eos. Two logical bitstreams are visible here by their serial numbers, both bos pages ahead of any content, and the two granule positions count in different units because the two codecs chose differently — 8000 Hz samples for the Vorbis stream and 48000 Hz for the Opus one.

How it is tested #

Three kinds of test, because each catches what the others cannot.

Against pages somebody else framed. src/tests/data/ holds three small Ogg files made with ffmpeg, and the tests assert their page tables field by field. This is what checks the page checksum: its parameters are not any of the catalogued CRC-32s, and the proof that they are right is that the value computed here is the one libogg wrote. A test that asserts what this implementation produced would prove only that it has not changed. src/tests/data/README.md says what each file is for and how it was made.

Against the specification's own statements. Where the format says something a test says it back: a nil page carries no packets, a packet whose length is a multiple of 255 ends in a zero lacing value, a lacing value of 255 means the packet runs on, a page where no packet finishes carries -1, a page is at most 65307 bytes.

Against itself, over every input. src/tests/fuzz.zig states four properties that must hold for any bytes at all: demuxing anything terminates, reports and frees; a page that parsed is written back unchanged; resynchronising accounts for every byte of the input, either as part of a page handed back or as one it says it dropped; and a round trip through the writer and the demuxer is the identity, at every page size and for packets of every length either side of a segment and a page boundary. Every target runs under an allocator with a budget, so the allocation-failure paths are exercised at the same time.

$ nix develop --command zig build test          # unit tests and the fuzz corpus
$ nix develop --command zig build check         # compile everything, run nothing
$ nix develop --command zig build fuzz --fuzz   # Zig's fuzzer, until interrupted
$ nix develop --command zig build fuzz-run -- --seconds 300
$ nix develop --command reuse lint
$ nix develop --command zig fmt --check .

zig build fuzz --fuzz runs those targets under Zig's own fuzzer, which follows coverage; --fuzz=1M makes it a bounded run that ends in a report. The fuzz test binary is compiled with LLVM even in Debug, because the self-hosted backend emits no coverage instrumentation and the fuzzer would have nothing to steer by. A finding prints input saved to '.zig-cache/f/crash' above the report.

zig build fuzz-run drives the same targets from tools/fuzz.zig instead, mutating a corpus of real streams rather than following coverage. It is the quicker of the two to reproduce a finding with (--input), and its findings land in fuzz-findings/.

Getting the source #

The repository has three homes, and they carry the same history.

The everyday one is a Forgejo instance:

git clone https://git.jcollie.dev/jeff/zig-ogg.git

A mirror is kept on Tangled, which builds its forge on the AT Protocol:

https://tangled.org/jcollie.dev/zig-ogg

It is also published on Radicle, a peer-to-peer code forge built on git, where a repository is found by its Repository ID (RID) rather than by a server name:

rad:z4ByfsGeREmwFqFkkBkn1AxsBs1Tj

If you do not have Radicle yet, install it, create an identity, and start your node:

curl -sSf https://radicle.dev/install | sh
rad auth
rad node start

Then clone the repository:

rad clone rad:z4ByfsGeREmwFqFkkBkn1AxsBs1Tj

rad clone consults your node's routing table to find seeds that carry the repository, and drops the working copy into a zig-ogg directory. Pass a target path as a second argument to put it somewhere else. If your node has not discovered a seed yet, name one directly with --seed <NID>.

The result is an ordinary git working copy on the main branch, with a rad remote wired up, so everyday work is just git. To pull in changes published since your clone:

rad sync --fetch
git pull rad main

Building #

It builds with Zig 0.17. The last release for Zig 0.16 is tagged v0.1.0, and the zig-0.16 branch carries it.

Zig is not on PATH; everything runs through the flake's devshell.

$ nix develop
$ zig build                       # the command-line tool into zig-out/bin
$ zig build docs                  # the API documentation into zig-out/docs
$ zig build docs-serve            # ...and read it at http://127.0.0.1:8000

The documentation has to be served rather than opened: what Zig emits is not a page but a program — a WebAssembly viewer, its javascript, and a tar of the sources it reads from — and a browser will not fetch those from a file:// page.

What is where #

File	What is in it
src/page.zig	The page header, the segment table, and the packet fragments a page carries. Pure: bytes in, bytes out, no I/O and no allocation.
src/crc.zig	The page checksum, and the note on why it is spelled out rather than taken from the standard library's catalogue.
src/reader.zig	Pages off a std.Io.Reader, and resynchronising after damage.
src/writer.zig	Packets into pages, one logical bitstream at a time.
src/demuxer.zig	Pages back into packets, any number of streams at once. The one part that allocates.
src/codec.zig	Which codec a logical bitstream's first packet says it is.
src/main.zig	The command-line tool, and the only thing here that opens a file.
tools/fuzz.zig	A corpus-mutating fuzzing loop around the same targets as Zig's fuzzer.
tools/docs_server.zig	A one-directory, loopback-only HTTP server for reading the docs.

Licence #

MIT. Every file carries an SPDX header and reuse lint is expected to pass; LICENSES/MIT.txt is the text. The Ogg streams under src/tests/data/ were generated for this project and are covered by REUSE.toml.

References cited #

Pfeiffer, Silvia. The Ogg Encapsulation Format Version 0. RFC 3533. RFC Editor, May 2003. https://www.rfc-editor.org/info/rfc3533.

Goncalves, Ivo, Silvia Pfeiffer, and Christopher Montgomery. Ogg Media Types. RFC 5334. RFC Editor, September 2008. https://www.rfc-editor.org/info/rfc5334.

Terriberry, Timothy B., Ron Lee, and Ralph Giles. Ogg Encapsulation for the Opus Audio Codec. RFC 7845. RFC Editor, April 2016. https://www.rfc-editor.org/info/rfc7845.

Xiph.Org Foundation. Ogg Logical Bitstream Framing. https://xiph.org/ogg/doc/framing.html. The statement of the checksum's parameters — "direct algorithm, initial val and final XOR = 0" — that RFC 3533 leaves out.

Xiph.Org Foundation. Ogg Mapping for FLAC. https://xiph.org/flac/ogg_mapping.html.

Xiph.Org Foundation. Vorbis I Specification. https://xiph.org/vorbis/doc/Vorbis_I_spec.html.

Xiph.Org Foundation. MIME Types and File Extensions. XiphWiki. https://wiki.xiph.org/MIMETypesCodecs. The collected identification packets of the published Ogg mappings, which is what src/codec.zig compares against.