about things notes.zzstoatzz.io
notes languages ziglang std io http-client.md
8.6 kB · 181 lines
Markdown
at main

std.http.Client #

checked against zig 0.16.0 and 0.17.0 unless a section says otherwise.

fetch streams the body into the writer you pass #

http.Client.fetch(.{ ..., .response_writer = w }) does not buffer the response and hand it back. it calls response.readerDecompressing(...).streamRemaining(w), writing body bytes to w as they arrive, so the writer decides the memory profile:

  • std.Io.Writer.Allocating holds the whole body in RAM: fine for small JSON, fatal for multi-GB downloads.
  • a std.Io.File.Writer streams to a file descriptor in flat memory.
  • a custom std.Io.Writer with an empty buffer and its own drain processes bytes as they arrive (split NDJSON on \n, say).
const file = try std.Io.Dir.createFileAbsolute(io, path, .{ .truncate = true });
var buf: [64 * 1024]u8 = undefined;
var fw = std.Io.File.Writer.init(file, io, &buf);
const res = try client.fetch(.{
    .location = .{ .url = url },
    .method = .GET,
    .headers = .{ .authorization = .{ .override = "Bearer ..." } },
    .response_writer = &fw.interface,
    .keep_alive = false,
});
fw.interface.flush() catch {};
file.close(io);
if (res.status != .ok) return error.HttpError;

the client itself is .{ .allocator = allocator, .io = io } with deinit(). download to <path>.part and std.Io.Dir.renameAbsolute it into place, so a partial write is never mistaken for a complete file.

the allocating writer keeps ownership #

aw.written() borrows the body and leaves aw.deinit() responsible for it. aw.toArrayList() and toOwnedSlice() transfer ownership and reset the writer to empty, so a defer aw.deinit() beside them frees nothing and the body leaks unless the new owner frees it. on a long-running process that is a leak per call: about 80 KB per HTTP call exhausted a 256 MB machine in about 25 minutes.

the low-level request API does not decompress #

client.fetch(...) decodes Content-Encoding. client.request(...) followed by response.reader(...) and streamRemaining does not; only response.readerDecompressing(...) does. the symptom of reading the wrong one is a JSON SyntaxError on a body starting 1f 8b.

to ask for identity encoding, use the typed header slot. extra_headers is additive, so the client still sends its own accept-encoding (gzip and deflate in 0.16.0 and 0.17.0) beside yours, and the server picks gzip:

var req = try client.request(.POST, uri, .{
    .headers = .{ .accept_encoding = .{ .override = "identity" } },
});

against a PDS, extra_headers alone put gzip, deflate… and a second accept-encoding: identity on the wire and got gzip back; the override sent only identity and got an unencoded body.

raw TLS reads (0.15) #

reading from std.crypto.tls.Client directly, stream returning 0 does not mean end of stream: TLS consumed input without producing plaintext yet (a partial record, or a handshake message). keep reading:

outer: while (total_read < response_buf.len) {
    var w: std.Io.Writer = .fixed(response_buf[total_read..]);
    while (true) {
        const n = tls_client.reader.stream(&w, .limited(remaining)) catch break :outer;
        if (n != 0) {
            total_read += n;
            break;
        }
    }
}

a custom writer for streaming parse #

a writer with an empty buffer sends every write, including zero-length ones, straight to drain:

const Parser = struct {
    writer: std.Io.Writer,

    fn init() Parser {
        return .{ .writer = .{ .vtable = &vtable, .buffer = &.{} } };
    }
    const vtable: std.Io.Writer.VTable = .{
        .drain = drainFn,
        .flush = std.Io.Writer.defaultFlush,
        .rebase = std.Io.Writer.failingRebase,
    };
    fn drainFn(w: *std.Io.Writer, data: []const []const u8, splat: usize) std.Io.Writer.Error!usize {
        const self: *Parser = @fieldParentPtr("writer", w);
        // `data` is a list of slices; the last one repeats `splat` times.
        // return how many bytes of `data` were consumed.
    }
};

a redirect chain that panics (0.16.0) #

fetch(... .method = .HEAD) against a three-hop redirect chain that crosses domains and ends in a relative Location can panic with attempt to use null value on req.connection.? in the inlined sendBodiless/receiveHead path (ReleaseSafe on x86_64-linux; macOS Debug hangs instead). redirect() releases the old connection and nulls r.connection before connecting again; when that connect fails partway, the request is left with no connection, the error is swallowed, and the next use crashes. a panic cannot be caught, so catch does not help.

HEAD https://a.example/<rkey>
  → 307 https://b.example/auth_callback?payload=...  (cross-domain)
  → 307 https://a.example/receive_auth_callback?...  (back)
  → 307 /<rkey>?auth_completed=true                  (relative)
  → 200

a bodiless HEAD or GET follows at most three redirects by default, so this chain sits exactly at the limit. not re-tested on 0.17. workarounds: disable the call behind a flag, shell out to curl -I --max-redirs 5, or send the HEAD with .redirect_behavior = .unhandled and accept the first hop's status.

long-lived clients and certificate rotation #

std.http.Client.now is set by the first HTTPS request, when the client scans the trust store, and every later TLS handshake validates certificates against that cached time. a server certificate rotated in with a notBefore after it fails validation for the life of the client, though the system clock is right and a fresh client succeeds. what surfaces may be only TlsInitializationFailed.

refresh now before HTTPS requests. a client whose now is already set skips the trust-store scan, so load the CA bundle explicitly first. TLS setup reads now outside the CA-bundle lock, so on a shared client serialize writes to it through connection setup, and release before waiting on headers or bodies so independent requests still progress. connections opened through connectTcpOptions need trust and time set the same way.

a regression test can issue a trusted certificate, set the client's cached time two days earlier, and request through both the ordinary and the explicit-address paths: the fixed client accepts the valid certificate and still rejects a hostname mismatch, without moving the system clock or turning verification off.

certificate trust and platform revocation #

zig loads the platform's certificates but not its verification policy. on macOS, Certificate.Bundle.rescanMac parses the system keychain files itself instead of asking Apple's trust evaluator, so a native client rejecting a revoked intermediate says nothing about a zig client. on linux, rescanLinux reads the system PEM bundle, which in a container is the image's file: updating the host's CA package changes nothing, and a client holding a loaded bundle needs a restart.

test distrust through the application's own HTTP and WebSocket transports, with a revoked chain, a valid control, and any alternate cross-sign; a curl probe tests curl's trust path, not zig's. the compiler's package fetcher imports the same loader, so a patched loader needs zig fetch re-checked too. an archive or Content-Type error there means TLS already succeeded, so the negative control has to fail during certificate validation.

sources #

  • zig 0.16.0 and 0.17.0 lib/std/http/Client.zig (fetch, now, request, redirect, Connection.Tls.create), lib/std/crypto/Certificate/Bundle.zig and Bundle/macos.zig, 0.17.0 lib/compiler/Maker.zig
  • pub-search backend/: reconciler HEAD check and the redirect panic
  • zat src/internal/xrpc.zig and embed-on-pds backend/src/oauth.zig (pdsAuthedRequest): the accept-encoding override checked against a PDS, 2026-04-09; zat v0.2.14 (commit 819dffe): the toArrayList leak
  • music-atmosphere-feed src/http.zig: raw TLS reads on 0.15
  • zat scripts/test_transport_tls_clock.py (commit c8a4cf0) and zttp scripts/test_tls_clock.py: rotation and concurrent requests over real TLS on macOS and Linux, 2026-10-14
  • ~/tangled.org/zat.dev/emudhra-audit-2026-10-21.md and zat.dev/security/emudhra-2026-10-21/zig-results.md (local): zttp accepted both published compromised-chain hosts on macOS where curl rejected them; patched-loader fetch and std.http controls, 2026-10-21. eMudhra advisory and test chains