# zig-ogg A reader and writer for the Ogg container format, written in Zig against [RFC 3533][rfc3533], the specification of the format. It is a library first: it has no filesystem, no threads and no dependencies, reads from a `std.Io.Reader` and writes to a `std.Io.Writer`, and leaves the question of where the bytes come from to whoever is calling it. A command-line tool comes with it, mostly to prove the library can be used. ```console $ zig-ogg info song.ogg song.ogg: 412 pages, 1 logical bitstream serial 0x784d00be Vorbis audio 2104 packets, 3015291 bytes, ends at granule 2646000 $ zig-ogg check song.ogg song.ogg: OK, 412 pages, 2104 packets ``` The API documentation is generated from the doc comments, where most of the explanation lives, and is published at . ## What Ogg is, and what it is not Ogg is framing and nothing else. It takes packets from a codec, chops them into pages of a few kilobytes with a checksum and a position marker on each, and interleaves the pages of however many logical bitstreams share the file. It knows nothing about what the packets mean — not even how long a second is, since "Ogg does not have a concept of 'time': it only knows about sequentially increasing, unitless position markers" (RFC 3533, Section 4). So this library moves packets in and out of pages and leaves every question about their contents to whoever wrote them. It does not decode Vorbis, FLAC, Opus or anything else; what it will tell you is which of them a stream claims to be, because that is written in the first page and is the container's business. Ogg files are `application/ogg`, `audio/ogg` or `video/ogg` depending on what they carry, per [RFC 5334][rfc5334]; `.ogg`, `.oga`, `.ogv` and `.opus` are the extensions in use. ## What it does **Pages** are parsed and written in full: the capture pattern, the version, all three header flags, the granule position with its `-1` sentinel, the serial and sequence numbers, the checksum, and the segment table with the lacing rules that make a packet's length unbounded. A page that parsed is written back byte for byte. **Demultiplexing** handles both kinds of multiplexing RFC 3533 defines, and does not need to be told which it is looking at. "Grouping" interleaves the pages of concurrent logical bitstreams; "chaining" concatenates whole streams end to end; both are just pages keyed by serial number, so a demuxer that looks the number up and reassembles per stream handles either, or the two together. **Recovering from damage** is what the capture pattern and the per-page checksum are for, and `PageReader.resync` uses them: it steps over corrupt bytes until a page whose checksum is right turns up, and says how many it threw away. A stream joined in the middle, a page lost from the sequence, and a page that contradicts the one before it about whether a packet was still open are all reported rather than quietly papered over — the packet that follows arrives with `after_hole` set. **Writing** puts packets into pages, choosing page boundaries by a target size, and gets the parts right that are easy to get wrong: the end-of-stream flag lands on a page with content on it, the beginning-of-stream page carries the identification packet alone, a page where no packet finishes carries no granule position, and a packet whose length is a multiple of 255 gets its terminating zero. Not implemented: seeking. The granule positions and the page index are all there, and bisecting on them is how seeking in Ogg is done, but nothing here does it yet — it needs a seekable input, which the `std.Io.Reader` this takes is not required to be. Nor is there a muxer that decides *for* you how to interleave several streams' pages; writing two streams into one file is done by pointing two `StreamWriter`s at the same output, which leaves the interleaving policy where the caller can see it. ## Using the library Fetch it and import the module it exposes as `ogg`: ```console $ zig fetch --save git+https://git.jcollie.dev/jeff/zig-ogg.git ``` ```zig const ogg = b.dependency("ogg", .{ .target = target, .optimize = optimize }); exe.root_module.addImport("ogg", ogg.module("ogg")); ``` ### Reading packets `Demuxer` is the whole of demultiplexing. Packets come out in the order the pages carrying them appear, each tagged with the serial number of the logical bitstream it belongs to: ```zig const ogg = @import("ogg"); var buffer: [ogg.Page.max_size]u8 = undefined; var d: ogg.Demuxer = .init(gpa, reader, &buffer, .{}); defer d.deinit(); while (try d.next()) |packet| { if (packet.first) { // The first packet of a stream identifies its codec. const codec = ogg.codec.identify(packet.data) orelse continue; std.debug.print("0x{x}: {s}\n", .{ packet.serial_number, codec.description() }); } // packet.data is valid until the next call to next(). } ``` A `Packet` carries a little more than its bytes: | Field | What it says | | --- | --- | | `serial_number` | Which logical bitstream it came from. | | `data` | The packet. Valid until the next call to `next`. | | `granule_position` | Where the codec's position counter stood afterwards, when the stream said — which is only for the last packet to finish on a page, since a page carries one position and that is whose it is. Null otherwise. | | `index` | Counts packets within the logical bitstream from zero. | | `first`, `last` | It came off the bos page, or finished on the eos page. | | `after_hole` | Something before this packet was lost. The packet itself is whole. | ### Writing packets One `StreamWriter` is one logical bitstream: ```zig var body: [ogg.Page.max_body_size]u8 = undefined; var w: ogg.StreamWriter = .init(writer, &body, .{ .serial_number = serial }); try w.writePacket(identification_header, 0); try w.writePacket(comment_header, 0); for (audio[0 .. audio.len - 1]) |frame| try w.writePacket(frame.bytes, frame.samples); try w.writeLastPacket(audio[audio.len - 1].bytes, total_samples); ``` `writeLastPacket` is worth using wherever the last packet is known, so that the end-of-stream flag goes on the page carrying it rather than on an empty page after it. `flush` ends the current page early, which is what a live stream wants and what the mapping's rule about the bos page needs; `finish` ends the stream. Two streams in one file — grouping — is two writers pointed at the same output. The format requires every bos page to come before any other page, which is why the writer does not hold those two back: ```zig var a: ogg.StreamWriter = .init(writer, &body_a, .{ .serial_number = 100 }); var b: ogg.StreamWriter = .init(writer, &body_b, .{ .serial_number = 200 }); try a.writePacket(a_header, 0); // both bos pages go out at once... try b.writePacket(b_header, 0); // ...and only then does content follow, in whatever order suits. ``` One stream after another — chaining — needs nothing at all: `finish` one and start the next. ### Pages on their own Below the demuxer, `PageReader` hands back one `Page` at a time for a caller that wants the framing rather than the contents, and `Page.parse` is the parser by itself, over bytes, with no I/O: ```zig var pr: ogg.PageReader = .init(reader, &buffer); while (try pr.next()) |page| { // page.lacing, page.body, page.packets() ... } ``` `Page.packets()` walks the packet fragments a page holds, saying of each whether it is the tail of one that began earlier and whether it ends here. That is the layer `Demuxer` is built on, and is what to use to do something else with. ### What allocates, and what does not **A page is bounded and a packet is not**, and that one fact decides the shape of everything here. A segment table holds at most 255 entries of at most 255 bytes, so no page exceeds 65307 bytes however hostile the stream is. `Page`, `PageReader` and `StreamWriter` therefore allocate nothing at all: each works out of one buffer the caller hands it, sized by the `Page.max_size` and `Page.max_body_size` constants, and a page read is handed back as slices into that buffer rather than copied out of it. A packet, on the other hand, can be continued from page to page without end. So `Demuxer` — the only part that reassembles one — takes an allocator, and takes a `max_packet_size` to refuse at, because otherwise a stream whose every page ends mid-packet is a request for all the memory there is. `max_streams` is the same answer to a stream that names a new serial number on every page. Even there, nothing is copied in the ordinary case: a packet that begins and ends inside one page points into the page buffer. ## The command-line tool ```console $ zig-ogg info ... # what logical bitstreams a file holds $ zig-ogg pages # one line per page $ zig-ogg packets # one line per packet $ zig-ogg check ... # verify every page's checksum and order ``` `pages` is the one to reach for when a file is behaving oddly, since it shows the framing decisions another muxer made: ```console $ zig-ogg pages src/tests/data/grouped.ogg 0 serial 0x6a32f4ce page 0 -b- 1 segments, 30 bytes, granule 0 58 serial 0xcf6f1915 page 0 -b- 1 segments, 19 bytes, granule 0 105 serial 0x6a32f4ce page 1 --- 11 segments, 2538 bytes, granule 0 2681 serial 0xcf6f1915 page 1 --- 1 segments, 60 bytes, granule 0 2769 serial 0x6a32f4ce page 2 --e 25 segments, 447 bytes, granule 6000 3268 serial 0xcf6f1915 page 2 --e 38 segments, 3194 bytes, granule 36312 ``` The three flag columns are `continued`, `bos` and `eos`. Two logical bitstreams are visible here by their serial numbers, both bos pages ahead of any content, and the two granule positions count in different units because the two codecs chose differently — 8000 Hz samples for the Vorbis stream and 48000 Hz for the Opus one. ## How it is tested Three kinds of test, because each catches what the others cannot. **Against pages somebody else framed.** `src/tests/data/` holds three small Ogg files made with ffmpeg, and the tests assert their page tables field by field. This is what checks the page checksum: its parameters are not any of the catalogued CRC-32s, and the proof that they are right is that the value computed here is the one libogg wrote. A test that asserts what this implementation produced would prove only that it has not changed. `src/tests/data/README.md` says what each file is for and how it was made. **Against the specification's own statements.** Where the format says something a test says it back: a nil page carries no packets, a packet whose length is a multiple of 255 ends in a zero lacing value, a lacing value of 255 means the packet runs on, a page where no packet finishes carries `-1`, a page is at most 65307 bytes. **Against itself, over every input.** `src/tests/fuzz.zig` states four properties that must hold for any bytes at all: demuxing anything terminates, reports and frees; a page that parsed is written back unchanged; resynchronising accounts for every byte of the input, either as part of a page handed back or as one it says it dropped; and a round trip through the writer and the demuxer is the identity, at every page size and for packets of every length either side of a segment and a page boundary. Every target runs under an allocator with a budget, so the allocation-failure paths are exercised at the same time. ```console $ nix develop --command zig build test # unit tests and the fuzz corpus $ nix develop --command zig build check # compile everything, run nothing $ nix develop --command zig build fuzz --fuzz # Zig's fuzzer, until interrupted $ nix develop --command zig build fuzz-run -- --seconds 300 $ nix develop --command reuse lint $ nix develop --command zig fmt --check . ``` `zig build fuzz --fuzz` runs those targets under Zig's own fuzzer, which follows coverage; `--fuzz=1M` makes it a bounded run that ends in a report. The fuzz test binary is compiled with LLVM even in Debug, because the self-hosted backend emits no coverage instrumentation and the fuzzer would have nothing to steer by. A finding prints `input saved to '.zig-cache/f/crash'` above the report. `zig build fuzz-run` drives the same targets from `tools/fuzz.zig` instead, mutating a corpus of real streams rather than following coverage. It is the quicker of the two to reproduce a finding with (`--input`), and its findings land in `fuzz-findings/`. ## Getting the source The repository has three homes, and they carry the same history. The everyday one is a Forgejo instance: ```console git clone https://git.jcollie.dev/jeff/zig-ogg.git ``` A mirror is kept on [Tangled](https://tangled.org/), which builds its forge on the AT Protocol: It is also published on [Radicle](https://radicle.dev/), a peer-to-peer code forge built on git, where a repository is found by its Repository ID (RID) rather than by a server name: ``` rad:z4ByfsGeREmwFqFkkBkn1AxsBs1Tj ``` If you do not have Radicle yet, install it, create an identity, and start your node: ```console curl -sSf https://radicle.dev/install | sh rad auth rad node start ``` Then clone the repository: ```console rad clone rad:z4ByfsGeREmwFqFkkBkn1AxsBs1Tj ``` `rad clone` consults your node's routing table to find seeds that carry the repository, and drops the working copy into a `zig-ogg` directory. Pass a target path as a second argument to put it somewhere else. If your node has not discovered a seed yet, name one directly with `--seed `. The result is an ordinary git working copy on the `main` branch, with a `rad` remote wired up, so everyday work is just git. To pull in changes published since your clone: ```console rad sync --fetch git pull rad main ``` ## Building It builds with Zig 0.17. The last release for Zig 0.16 is tagged `v0.1.0`, and the `zig-0.16` branch carries it. Zig is not on `PATH`; everything runs through the flake's devshell. ```console $ nix develop $ zig build # the command-line tool into zig-out/bin $ zig build docs # the API documentation into zig-out/docs $ zig build docs-serve # ...and read it at http://127.0.0.1:8000 ``` The documentation has to be served rather than opened: what Zig emits is not a page but a program — a WebAssembly viewer, its javascript, and a tar of the sources it reads from — and a browser will not fetch those from a `file://` page. ## What is where | File | What is in it | | --- | --- | | `src/page.zig` | The page header, the segment table, and the packet fragments a page carries. Pure: bytes in, bytes out, no I/O and no allocation. | | `src/crc.zig` | The page checksum, and the note on why it is spelled out rather than taken from the standard library's catalogue. | | `src/reader.zig` | Pages off a `std.Io.Reader`, and resynchronising after damage. | | `src/writer.zig` | Packets into pages, one logical bitstream at a time. | | `src/demuxer.zig` | Pages back into packets, any number of streams at once. The one part that allocates. | | `src/codec.zig` | Which codec a logical bitstream's first packet says it is. | | `src/main.zig` | The command-line tool, and the only thing here that opens a file. | | `tools/fuzz.zig` | A corpus-mutating fuzzing loop around the same targets as Zig's fuzzer. | | `tools/docs_server.zig` | A one-directory, loopback-only HTTP server for reading the docs. | ## Licence MIT. Every file carries an SPDX header and `reuse lint` is expected to pass; `LICENSES/MIT.txt` is the text. The Ogg streams under `src/tests/data/` were generated for this project and are covered by `REUSE.toml`. ## References cited Pfeiffer, Silvia. *The Ogg Encapsulation Format Version 0*. RFC 3533. RFC Editor, May 2003. . Goncalves, Ivo, Silvia Pfeiffer, and Christopher Montgomery. *Ogg Media Types*. RFC 5334. RFC Editor, September 2008. . Terriberry, Timothy B., Ron Lee, and Ralph Giles. *Ogg Encapsulation for the Opus Audio Codec*. RFC 7845. RFC Editor, April 2016. . Xiph.Org Foundation. *Ogg Logical Bitstream Framing*. . The statement of the checksum's parameters — "direct algorithm, initial val and final XOR = 0" — that RFC 3533 leaves out. Xiph.Org Foundation. *Ogg Mapping for FLAC*. . Xiph.Org Foundation. *Vorbis I Specification*. . Xiph.Org Foundation. *MIME Types and File Extensions*. XiphWiki. . The collected identification packets of the published Ogg mappings, which is what `src/codec.zig` compares against. [rfc3533]: https://www.rfc-editor.org/rfc/rfc3533.html [rfc5334]: https://www.rfc-editor.org/rfc/rfc5334.html