zig-http as a client #
Making requests with zig-http: http_io.Client, which takes a URL and works out
the rest, and underneath it the three drivers that each speak one version of
HTTP over a connection handed to them. The API documentation at
https://jeff.jcollie.page/zig-http/ is generated from the doc comments; this
is the account of how the pieces behave together, and of what testing them
against other implementations found.
The server side is server.md.
Fetching a URL #
http_io.Client is for a caller holding a URL rather than a connection. It
has the shape of std.http.Client.fetch — a location, a method, an optional
payload, a writer for the body, a status back — and does the parts the drivers
below leave to whoever holds them: it parses the URL with
zig-uri, resolves the name, dials,
negotiates the version, and follows redirects.
var client: http_io.Client = .{ .io = io, .gpa = gpa };
defer client.deinit();
var body: std.Io.Writer.Allocating = .init(gpa);
defer body.deinit();
var result = try client.fetch(.{
.location = .{ .url = "https://example.com/" },
.response_writer = &body.writer,
});
defer result.deinit();
// result.status == .ok, result.version == .http_2,
// result.headers.get("content-type").? == "text/html"
The version is the server's choice. An https URL is dialled offering
h2 ahead of http/1.1 in ALPN, and whichever the server selects is the
driver that speaks — §3.2 of RFC 9113 makes that the only way HTTP/2 over TLS
is agreed. A server that answers with no ALPN extension at all is not taking
part in the negotiation (§3.1 of RFC 7301 makes answering optional), and gets
HTTP/1.1, which is what an https URL meant before there was anything to
negotiate. Cleartext has no handshake to negotiate in and RFC 9113 deprecated
the Upgrade that used to stand in for one, so an http URL is HTTP/1.1
unless http2_prior_knowledge says the server speaks HTTP/2 from the first
octet. http2 = false offers http/1.1 alone.
The name is resolved through std.Io.net.HostName.lookup, or through
Client.resolver when one is given, and the addresses are tried in the order
the resolver gave until one accepts the connection — tcp.connectAny and tls.connectAny, which are there for any
caller with a list of addresses. A certificate that fails to verify at one
address is not retried at the next: every address a name resolves to is meant
to be the same server, and trying them until one says something different is
the shape of an attack rather than a fallback. Host names are ASCII, so an
internationalised one has to arrive in its xn-- form.
The target is sent as it was written. zig-uri keeps a path and a query
decoded, and decoding forgets which reserved characters were escaped: a%2Bb
and a+b both come back as a+b. §2.2 of RFC 3986 says those are different
URIs, and a server that reads a query as a form — Django, PHP, Rails, Go's
ParseQuery — takes a bare + for a space. So the path and query of a .url,
and of a Location that is followed, go out exactly as written, after a check
that they read back as the URI the client resolved. A target that is not
written that way already — a space in it, an IRI, a dot segment that resolution
removed — and a .uri location, which has no written form, are encoded again
from the parsed URI.
The body can be routed by the status. response_writer gets every body
whatever its status, a 404's error page as much as a 200's content.
response_writer_for is the alternative for a caller who cares — a download
going into a file, say: its choose is called once with the status, version
and fields of the response fetch will return, before any of its body is
read, and returns the writer for the body or null to discard it.
const OnSuccess = struct {
file: *std.Io.Writer,
fn choose(context: *anyopaque, head: http_io.Client.ResponseHead) std.Io.Writer.Error!?*std.Io.Writer {
const self: *@This() = @ptrCast(@alignCast(context));
return if (head.status.class() == .success) self.file else null;
}
};
var on_success: OnSuccess = .{ .file = &file_writer.interface };
var result = try client.fetch(.{
.location = .{ .url = "https://example.com/big.iso" },
.response_writer_for = .{ .context = &on_success, .choose = OnSuccess.choose },
});
The certificate is checked against the system's store unless root_ca
names a bundle. That is the opposite default from tls.ClientOptions, which
trusts nothing until told, and deliberately so: that is a dialler handed an
address by a caller who has already decided what it is talking to, and this is
what a caller reaches for when it has a URL and wants what is at the other end.
Redirects are followed as std.http.Client follows them: up to three for a
request without a payload and none for one with, since resending a body to
wherever a server points is the caller's decision — redirects = .{ .follow = n } makes it. Location is resolved against the URL that was fetched by RFC
3986 §5's algorithm, a 303 becomes a GET (§15.4.4 of RFC 9110), a POST
answered 301 or 302 becomes one too (§15.4.2 and §15.4.3 allow it and every
browser does it), and a 307 or 308 keeps the method and the content.
Headers come in two kinds: extra_headers go wherever the request goes, and
privileged_headers — an Authorization, a Cookie — stop at the first
redirect to another origin, the scheme, host and port triple of §4.3.1 of RFC
9110. A URL carrying userinfo is refused outright, as §4.2.4 of RFC 9110 asks
of a recipient; a credential belongs in privileged_headers, where a redirect
cannot carry it off. A redirect from https to http is followed, as browsers,
curl and std.http.Client follow it, unless refuse_insecure_redirects is
set — then it is error.InsecureRedirect, before anything goes out in
cleartext, since whatever follows a downgrade crosses the network readable by
anybody on the path, who may also be the one who asked for it.
What it does not do is keep anything between calls. Every fetch dials,
sends, reads and closes, so HTTP/2's multiplexing is not used across calls and
a redirect to the same origin opens a second connection; a caller wanting many
requests on one connection uses the drivers directly, which is what they are
for. A Client is not safe to share
between tasks — it holds a random number generator and a lazily loaded
certificate bundle, so one each is cheap. Over HTTP/2 the body is bounded by
http2_config.max_body_len, because that driver gathers a body before handing
it over; over HTTP/1.1 it streams into the writer.
HTTP/3 is asked for, not negotiated. Over TCP, ALPN chooses the version
inside the one handshake; QUIC is another transport on another protocol, so
trying it is a decision made before anything is sent. http3 = .only fetches
an https URL over HTTP/3 or fails with the handshake's error, and .prefer
tries QUIC first and falls back to TCP — before anything was sent, so a
request is never sent twice. Each address the name resolved to is tried in
turn, and a handshake is given http3_handshake_timeout, three seconds by
default: the usual reason for no answer is UDP blocked on the way, which is
silence rather than a refusal and costs the whole wait. A certificate refused
over QUIC is never retried at another address or over TCP, as over TLS. An
http URL is never fetched over HTTP/3, since RFC 9114 has no cleartext form,
and neither is anything going through a proxy, which would be RFC 9298's
CONNECT-UDP. Against the internet: Cloudflare, Google and this repository's
Forgejo answer over HTTP/3, and example.com, which has none, fails .only and
falls back under .prefer.
Or HTTP/3 where the origin has said so, which is http3 = .advertised and
how a browser finds it. RFC 7838's Alt-Svc: h3=":443"; ma=86400 on any
response says the origin can be had over QUIC there for the next day; the
first fetch to an origin goes over TCP, since nothing has been said yet, and
what its response advertises is remembered in an http_io.AltSvc — a fixed
table of 32 origins, allocating nothing, with a lock so that Clients on
several tasks can share one through Client.alt_svc and what any of them
learns the rest use. Later fetches go to the alternative until it stops being
fresh: ma, less the response's Age, as §3.1 does the arithmetic. A new
Alt-Svc replaces what the origin said before, clear forgets it, and only
h3 is kept — the drafts' h3-29, which Google still sends, is skipped for
the h3 beside it. The alternative may name another host, and the
connection goes there, but SNI and the certificate check are the origin's —
§2.1's "reasonable assurances that the alternative service is under control
of and valid for the whole origin" are exactly a certificate for it. So the
cache learns only from responses over TLS made directly: not over cleartext,
where anybody could have said it, and not through a proxy, which §2.4 says a
request should keep going through. An alternative that fails falls back to TCP
before anything was sent and is left alone for five minutes, though every
response goes on advertising it, so that a network dropping UDP costs one
handshake timeout rather than one per fetch. AltSvc.clear forgets everything,
which §2.2 wants when the network changes and which only the caller can see.
And HTTP/3 from DNS, which Alt-Svc cannot do: an Alt-Svc arrives on a
response, so the first request to an origin has already gone over TCP. RFC
9460's HTTPS records say the same thing in DNS before anything is sent, and
Client.services is a hook — HostName.lookup's shape, as resolver is —
through which .advertised asks for them. The standard library cannot ask for
record type 65, and zig-dns-client, which can, cannot be a dependency of
http_io, so the program supplies it. The hook does SVCB resolution — query
names, aliases, mandatory keys — and hands back endpoints; Client tries
them in priority order, over QUIC where an endpoint's ALPN set holds h3 and
over TLS where it holds anything that runs over TCP, with the origin's name for
SNI and the certificate wherever the connection goes, and falls back to the
origin when none answers. A cached Alt-Svc is tried before the records. A
lookup that fails over DNS the hook knows to be protected abandons the fetch,
since carrying on without the records is the downgrade §3.1 warns of; nothing
is asked through a proxy (§3.2). And an http URL whose origin has a record is
fetched as https, as §9.5 asks — https_record_upgrade turns that off.
https-records.md is the plan this came from.
zig-dns-client's ServiceLookup is the resolver's half, and
interop/https-records/ is where the two meet: a project of its own, with a
build.zig.zon of its own that depends on zig-http by path and on
zig-dns-client by URL, so that neither this build nor any project depending on
zig-http ever fetches zig-dns-client. Its https_records module is the glue a
program using both needs — Client.Service and zig-dns-client's Service are
separate types, since each library pins its own copy of the other, so the
endpoints are copied across — and its test runs a DNS server on loopback
publishing HTTPS records, zig-http serving over TCP and QUIC, and a client that
takes its very first request to HTTP/3 on a port only DNS named, follows an
alias, upgrades an http URL, and stays on TCP for a name with no records:
$ cd interop/https-records
$ nix develop ../.. -c zig build test --summary all
A body too large to hold goes in payload_reader rather than
payload: a reader and a length. The length is declared as the
Content-Length, and exactly that many octets are then read from the reader
straight into the connection — after the head over HTTP/1.1, and over HTTP/2 as
DATA frames sized by both flow-control windows — with nothing buffered beyond
the connection's own buffers, so a 32 MiB upload in tests/fetch.zig runs in
15 MB of memory. A reader that ends before the length is
error.RequestBodyShort and one that fails is error.RequestBodyReadFailed,
never a shorter body: over HTTP/1.1 the connection is abandoned mid-message,
and over HTTP/2 the stream is reset with CANCEL so the server stops waiting.
A reader is read once, so a redirect that would send the body again — a 307
or 308, or a 301 or 302 to anything but a POST — is
error.StreamedPayloadRedirected, while a 303, which is answered with a
GET, is still followed. Underneath it is client.Request.source, which all
three drivers take: HTTP/3 sends it as DATA frames of sixteen kibibytes, each
filled before its header is written, with QUIC's flow control deciding the
pace, and resets a stream it cannot finish with H3_REQUEST_CANCELLED. That
last part found a bug in zig-quic — a stream reset while blocked on flow
control declared a final size larger than the peer had allowed, and the peer
closed the whole connection — which is fixed at the pin this uses.
A timeout is Client.timeout, a std.Io.Timeout — a duration or a
deadline — and it is one budget for the whole fetch: the lookup, the
connection, the handshake, every redirect and all of the body. Not one per
read, because a server trickling a byte a second never trips one of those, and
because the standard library has none to give portably: Zig 0.17 times a
socket read only on Io.Threaded, panicking on its other Ios, and its
connect timeout is a @panic("TODO"). So a timed fetch runs as
a task of its own beside one that sleeps until the deadline, std.Io.Select
reports whichever finishes first, and the other is cancelled — which
interrupts a blocked read at once rather than at the next octet, and closes the
connection on the way out like any other failure. It is error.Timeout, and it
needs an Io that can run two tasks together; zig-http --fetch URL --fetch-timeout 30 sets one.
result.headers are the last response's fields — the one that was not
followed — in the order they arrived, with a repeated field kept as the
separate lines it came as, and looked up without case, since HTTP/2 sends
every name in lowercase and HTTP/1.1 however the server wrote it. They are
copied out of the connection, which fetch closes before it returns, so a
result owns them and deinit is what frees them. This is where FetchResult
parts company with std.http.Client's, which has a status and nothing to
free.
result.status is std.http.Status, as std.http.Client.FetchResult has it:
non-exhaustive, so a code nothing has a name for still arrives, and class()
is what §15 of RFC 9110 has a client act on for one. It is never an interim
response's: a 100 Continue or a 103 Early Hints ahead of the answer is read
past, and options.interim is how to see them on the way — see
Interim responses below. Over a redirect it is told of
every hop's, in the order they came.
Client.resolver is a function with std.Io.net.HostName.lookup's signature,
and that function is its default — so anything written to stand in for the
standard library's lookup stands in for it here too. The contract is that
function's: addresses go into the queue carrying the port asked for, the queue
is closed on return even on error, and a queue of sixteen must not block.
There is no context pointer, so a replacement that needs configuration keeps it
at container level.
The demonstration binary uses the default.
Through a proxy #
http_proxy, https_proxy and no_proxy on Client, and
initDefaultProxies to fill them from the environment the way curl and
std.http.Client read it. A Proxy is a host, a port, whether the connection
to it is TLS, and a Proxy-Authorization value; Proxy.parse makes one out of
http://user:secret@proxy:3128, turning the userinfo into a Basic credential
as RFC 7617 spells it, and takes a value with no scheme as http, which is
how a good many environments write it.
Which way through a proxy goes depends on what is being fetched:
- An
httpURL is forwarded. The request goes to the proxy with its target in absolute form —GET http://origin/ HTTP/1.1, §3.2.2 of RFC 9112 — and the proxy fetches it. It is HTTP/1.1 to the proxy whatever the origin speaks, and a407is simply the response, since it came from the only server this end spoke to. - An
httpsURL is tunnelled.CONNECT origin:443asks the proxy for a byte stream (§9.3.6 of RFC 9110), and the TLS handshake runs inside it. So the certificate checked is the origin's, the proxy sees ciphertext and nothing else, and ALPN reaches the origin through the tunnel and comes backh2exactly as it would without one. A tunnel that is refused is an error rather than a response —error.ProxyAuthenticationRequiredfor a407,error.ProxyRefusedfor anything else — because nothing a proxy says in answer toCONNECTcame from the origin.
The connection to the proxy is either plain or TLS, independently of both: an
https URL through an https:// proxy is TLS inside TLS, which is why the
TLS session's own read buffer is one whole record rather than a round 16 KiB.
The origin's name is resolved by the proxy and never by this end, which is
half the reason proxies are used.
The pieces are public for a caller holding the drivers rather than Client:
http_io.proxy.tunnel asks for a CONNECT over any transport.Connection
and hands the same connection back carrying the tunnel, and
http_io.tls.connectOver runs a handshake over a connection that is already
open. http_io.proxy's documentation has the four lines that put an HTTP/2
driver behind a proxy. What there is not is a way through a proxy for HTTP/3,
which would be RFC 9298's CONNECT-UDP.
zig-http --fetch URL --fetch-proxy http://user:secret@proxy:3128 uses one,
and without --fetch-proxy the environment's is used.
tests/nixos/proxy.nix puts tinyproxy in front of an origin whose
firewall admits nothing else, and asks both ways through it, with and without
the right credential.
Trying it #
zig-http --fetch URL, with no --fetch-version, is this client, and
tests/fetch.zig runs it against this library's own server over loopback
sockets, with and without TLS, looking localhost up for real.
A client, for HTTP/1.0 and HTTP/1.1 #
http.client is the mirror of http.server: a Request is what
server.Request describes seen from the other end, and a Response the same
for server.Response, so a caller who has written a handler already knows
both. http1.client.Connection puts them on a byte stream, over the same
transport.Connection the server is served over — which means it runs over
TCP, over TLS, and over a slice of memory in a test without knowing the
difference.
const conn = try http_io.tcp.connect(io, gpa, address, .{});
var c = try http.http1.client.Connection.init(io, gpa, conn, .{});
defer c.deinit();
const res = try c.send(.{ .host = "example.com", .target = "/" });
const body = try c.readAll(res, 1 << 20);
defer gpa.free(body);
Almost every rule in the driver is about framing, because framing is where an
HTTP/1.1 client is dangerous to itself. A response whose length the client
reads differently from the server that sent it ends in the wrong place, and
what follows it on a kept-alive connection is then read as the next
response — which is response splitting, done to oneself. So §6.3's eight rules
are applied in order by protocol.responseFraming, a message that frames
itself two ways at once is refused outright rather than resolved, and a
connection whose framing was uncertain is never reused. canReuse is the one
question a caller has to ask, and a body left unread answers it false.
Two things worth knowing about lifetimes. Every slice in a response — the
reason phrase, every field name and value — points into the connection's head
buffer, so a response is valid until the next send or until the connection
goes away. And deinit closes the connection it was given, the way the server
closes the one it accepted; release is the way out for a caller that wants
the socket afterwards, which is what a 101 leaves behind.
What a request says about how long its body is #
Over HTTP/1.1 a Content-Length is the only thing delimiting content, so the
driver has always written one. Over HTTP/2 and HTTP/3 it is optional — §8.1 of
RFC 9113 and §4.1 of RFC 9114 delimit the content with the stream — and both of
those drivers used to leave it out on that ground, which was a mistake in two
ways.
§8.6 of RFC 9110 asks for it: "A user agent SHOULD send Content-Length in a
request when the method defines a meaning for enclosed content and it is not
sending Transfer-Encoding." Recipients use it to check that what arrived is
what was sent, and a server that has it can read a body without waiting to find
out how long it is — where a recipient of an HTTP/3 body without one learns the
length only when it stops arriving, which is why req.body.length comes back
unknown there and none over HTTP/2. And a server may simply insist:
Cloudflare's DNS-over-HTTP/3 endpoint answers 400 to a POST without one,
which is how this was found — ten requests without the field failed, ten with it
did not.
So both multiplexed drivers now send it when the length is known, and
Request.declaredContentLength is the one place that decides what "known"
means, so the two cannot drift: a .exact body declares its length, a .none
body declares nothing (not Content-Length: 0 — a request with no body is not
a request with an empty one), and an .unknown one declares nothing either,
because the caller left the question open and a field filled in from whatever
is in hand would be the driver answering it.
It is also where the length is checked against the content in hand. A
Content-Length that disagrees with the DATA frames makes the message
malformed — §8.1.1 of RFC 9113 and §4.1.2 of RFC 9114 both say so — so a
request whose two halves disagree is error.InvalidRequest before anything goes
out, which is the same thing Request.reserved does for a caller trying to set
the field by hand.
HTTP/2, where a request is begun and awaited separately #
http2.client is the same idea over RFC 9113, and the difference is not the
plumbing but the shape of an exchange. HTTP/2 is multiplexed, so a request is
begun and awaited as two steps, and several may be outstanding between
them:
var c = try http.http2.client.Connection.init(io, gpa, conn, .{});
defer c.deinit();
try c.handshake();
const first = try c.begin(.{ .host = "example.com", .target = "/one" });
const second = try c.begin(.{ .host = "example.com", .target = "/two" });
const a = try c.await(first); // frames for `second` are filed as they arrive
const b = try c.await(second);
The pump is the caller's, not a task's. The server runs a reader task and
a task per stream, because it does not know when its next request arrives and
a handler must not block the connection. A client knows exactly what it asked
for, so nothing needs to run on its own: await reads frames until the stream
it was asked about is finished, filing frames for other streams into their own
slots as it goes. Two requests begun before either is awaited are genuinely in
flight together, and it costs no Io.Group, no mutex and no second task. What
it does cost is a buffer per outstanding stream, bounded by the window the
client advertises — which is what flow control is for.
Flow control is real on both sides: a request body larger than the 65535-octet
initial window and the 16 KiB frame limit is split and waits for
WINDOW_UPDATE, which tests/tcp.zig exercises with 200 KiB against this
library's own server.
Interim responses #
§15.2 of RFC 9110 lets a server send any number of 1xx responses ahead of the
final one, and requires that "a client MUST be able to parse one or more 1xx
responses received prior to a final response, even if the client does not
expect one". Every driver here reads them and returns the final response; none
of them is ever what send or await hands back. A caller that wants to see
them on the way — the Link fields of a 103 Early Hints, to start fetching
a stylesheet before the page is ready, which is what RFC 8297 is for — gives
the request a hook:
const Hints = struct {
fn told(context: ?*anyopaque, head: http.client.Request.InterimHead) void {
const self: *@This() = @ptrCast(@alignCast(context.?));
if (head.headers.get("link")) |link| self.preload(link);
}
// ...
};
var hints: Hints = .{};
const res = try c.send(.{
.host = "example.com",
.target = "/",
.interim = .{ .context = &hints, .call = Hints.told },
});
fetch takes the same thing as FetchOptions.interim. The head it is shown is
borrowed: its fields point into the connection and are gone once the hook
returns, so copy what has to outlive the call. Over HTTP/2 the hook is called
by whoever is reading the connection, which may be an await for a different
request on it — frames are filed as they arrive, whatever stream they are for
— so its context has to stay valid until that request's own response has been
returned.
A 101 Switching Protocols is not one of these. It is informational by number
and final in meaning: it ends HTTP on the connection, so over HTTP/1.1 it is
the response send returns, and over HTTP/2 and HTTP/3, which removed it, it
is MalformedResponse. So is an interim response that ends the stream, which
§8.1 of RFC 9113 and §4.1 of RFC 9114 both forbid.
§15.2 puts no number on them, so each driver's Config does:
max_interim_responses, sixteen by default, past which the request fails with
TooManyInterimResponses rather than waiting for ever on a server that only
ever says 100. Each one is let go of once it has been told — the HTTP/1.1
driver reads the next head into the same buffer, and the multiplexed ones start
the stream's arena again — so sixteen cost what one does.
Before the drivers knew about them, each got them wrong in its own way, and
the ways are worth recording. The HTTP/1.1 client returned the 103 as the
response and left the final one unread on the wire, on a connection it still
called reusable, so the next request on it was answered with this one's
response. The HTTP/2 client returned the 103 with the final response's
fields appended to it, a literal :status among them; it appended a trailer
section the same way, unchecked. The HTTP/3 client took the final response for
a trailer section and the body after it for a malformed message. The HTTP/2
client now tells a stream's HEADERS apart — interim, final, and at most one
trailer section, which ends the stream and is dropped as the other two drivers
drop theirs — and a malformed response on one stream is reported by that
stream's await rather than by whichever await happened to read it.
Holding the content back: Expect: 100-continue #
A large upload to a server that is going to refuse it — a 401 for a
credential that has expired, a 413 for a file too large — costs the whole
upload before the refusal arrives, and over HTTP/1.1 the only way to stop a
body once it has started is to close the connection. §10.1.1 of RFC 9110 is
the way round: the request says Expect: 100-continue and goes out without
its content, and the server answers the head alone — a 100 Continue to ask
for the content, or the final response that refuses it.
const res = try c.send(.{
.method = http.protocol.Method.put,
.host = "example.com",
.target = "/backups/disk.img",
.body = .{ .exact = size },
.source = &file_reader.interface,
.expect_continue = true,
});
if (!res.content_sent) {
// Refused before any of it went: `res.status` says why, and the reader
// has not been touched.
}
expect_continue is a field rather than a header — Expect is one of the
fields Request.reserved keeps for the driver — because the driver has to know
to wait, and a field set by hand would say one thing while the driver did
another. It is dropped where it cannot apply: a request with no content has
nothing to hold back, and HTTP/1.0 has no 100 to answer it with.
What happens once the head is out:
- A
100 Continue, and the content goes, and the final response is read as usual. The100is an interim response like any other:interimis told of it andmax_interim_responsescounts it. - A final response first, and the content never goes:
content_sentis false. Over HTTP/1.1 the connection is then not reused, since the server may be waiting for the content or may be about to close, and there is no telling which. Over HTTP/2 the request's half of the stream is closed with RST_STREAM ofNO_ERRORonce the response is complete, and over HTTP/3 with a reset ofH3_NO_ERROR— §8.1 of RFC 9113 and §4.1 of RFC 9114's way of saying the content is not coming, rather than ending the stream as though it had been empty.
Over HTTP/2, begin sends the head alone and the content goes out from that
request's own await once its 100 has arrived — even if some other request's
await was the one that read the 100. So the content slice or source
reader has to stay valid until then.
There is no timer. The usual practice — curl's — is to wait about a second
for the 100 and then send the content anyway, for the sake of a server that
ignores the field. Zig 0.17 times a socket read only on Io.Threaded and
never through TLS, so a driver here waits for as long as the server takes.
§10.1.1 has an origin server answer such a request "immediately", so a
conforming one never leaves it waiting; a server or an intermediary that
ignores the field leaves it waiting for ever. That is why it is something a
caller asks for and not something that happens by default.
fetch takes it as FetchOptions.expect_continue — .never, the default;
.always; or .{ .above = n } for a payload longer than n octets — and uses
timeout as the bound the drivers do not have, so the two belong together:
var result = try client.fetch(.{
.location = .{ .url = "https://example.com/upload" },
.method = http.protocol.Method.put,
.payload_reader = .{ .reader = &file_reader.interface, .length = size },
.expect_continue = .{ .above = 1 << 20 },
});
Two things follow from fetch owning the whole exchange. A redirect that
arrives before the content was asked for is followed with the content still
unread, so a 307 sends a payload_reader on to where it is wanted rather
than ending in error.StreamedPayloadRedirected. And a 417 Expectation Failed — what an intermediary that does not understand the field may answer —
is met by sending the request once more without the expectation, as §15.5.18
of RFC 9110 suggests.
Against a server this library did not write #
tests/nixos/client.nix points the client at nginx, which answers HTTP/1.1
and cleartext HTTP/2 on one port by looking at the first octets — the same
thing this library's own server does, so one guest gets a second opinion on
both — and HTTP/3 on another, with nginx's own QUIC stack. It fetches over
1.0, 1.1, 2 and 3, asks for a body larger than one frame and one window over
each, and checks that a 404 comes back as a 404.
It earned its place immediately. Two faults survived every in-process test and died here:
- The connection window is not the stream window. §6.9.2 starts the
connection's flow-control window at 65535 and
SETTINGS_INITIAL_WINDOW_SIZEdoes not change it — only aWINDOW_UPDATEon stream 0 does. The client accounted against a window nginx had never agreed to, so nginx stopped after 65535 octets and waited for credit while the client waited for the rest. Neither end was at fault, and against this library's own server it never showed, because both halves made the same assumption. - A frame payload is not necessarily in the buffer. §4.2 puts the floor
for
SETTINGS_MAX_FRAME_SIZEat 16384, which is larger than a connection's default read buffer, so a payload has to be streamed rather than taken whole. The server had always done this; the client had not, and would have worked against peers that send small frames and failed against those that do not.
The demonstration binary fetches as well as serves — --fetch URL with
--fetch-version 1.0, 1.1 or 2 — which is what the guest drives. Naming
a version means one driver pointed at one socket, so that form takes an
address rather than a name; IpAddress.resolve sounds like it would take a
hostname and does not, since it parses a literal and the "resolve" in the name
is about an IPv6 interface scope. Leaving the version out is
http_io.Client, which looks the name up with
HostName.lookup and lets the server choose the version — and the guest asks
it that too, by name, over cleartext and over TLS.
HTTP/3, where most of the work is absent #
http3.client is written against transport.Session and nothing else — the
same seam the server driver uses, which is what made it possible without a
second QUIC implementation. http_io.http3.Dialer dials and hands one over.
Most of what the HTTP/2 client does has no counterpart here, and every absence
is §4.6 giving the job to QUIC. No connection preface, because the handshake
is one. No stream table, because streams belong to the transport. No flow
control and no WINDOW_UPDATE — the two faults nginx found in the HTTP/2
client were both in code this file does not have. No shared decoder state to
lose, because §3.2.2 of RFC 9204 with a table capacity of zero means no field
section can depend on an insertion. What is left is: open the control stream,
say what the settings are, put a HEADERS frame and some DATA frames on a
bidirectional stream, and read the same shape back.
Concurrency is the caller's, for the same reason: several requests are several QUIC streams, which is what §4.6 hands to the transport.
A request ends with its last octets rather than behind them. transport.Session
has a finishWith beside its finish, and Stream.Writer.finish hands the
buffered tail to it instead of flushing first -- so a request goes out as one
STREAM frame with the FIN bit rather than as the frame and then an empty frame
carrying only the bit. §19.8 of RFC 9000 permits both and every other
implementation sends the first, which turned out to matter: Quad9's
DNS-over-HTTP/3 server answers a request of that shape and silently ignores one
of the other, acknowledging the packets either way. It was found by a DNS client
built on this library, and the transport half of the fix is in zig-quic.
One thing to know about the session underneath — quic_io.Client.run wants a
task of its own. A stream read blocks until a datagram arrives and the pump is
what makes one arrive, so a caller doing both on one task waits for itself.
http3.client.Connection.run wants one too, for the streams the server opens:
its control stream, which lasts as long as the connection, and its two QPACK
streams. Responses arrive without it, on the streams send opened, but the
server's GOAWAY arrives only through it — after one, send refuses with
error.GoingAway rather than open a request the server has said it will not
process — and so do the §6.2 and §7.2 rules a server can break on those
streams, each of which closes the connection. http_io.Client runs it beside
the pump.
The demonstration binary fetches over all four versions:
$ zig-http --fetch https://127.0.0.1:8443/ --fetch-version 3
200
content-type: text/plain; charset=utf-8
tests/nixos/h3client.nix points it at aioquic's HTTP/3 server, which is
aioquic.nix turned around: that one puts aioquic's client in front of this
server, and says what this library writes can be read by somebody else.
What it can read is a separate claim, and the one the HTTP/2 client failed
until nginx was asked.
The paths are chosen for where a client goes wrong rather than for coverage: a 204 with no body at all, where a client waiting for a DATA frame stops for ever; a response larger than one stream window; a second field section after the content, which §4.1 allows and which a client must consume rather than mistake for another response; and a field section too large for one packet, which is read whole or read wrong — a failure that depends on timing, so it is absent on a loopback socket and present here.
tests/nixos/traefik.nix asks two further things. Traefik's HTTP/3 is
quic-go, a third independent QUIC implementation after nginx's own and
aioquic's. And Traefik is a proxy, not an origin: it reads the request over
HTTP/3, writes it again as HTTP/1.1 to static-web-server, reads that answer
and writes it back — so every field in the response has been parsed and
re-serialised by something in the middle, which is where a length, a coding or
a connection-specific field goes wrong, and is the shape almost every real
deployment has.
Nothing was found, which is worth saying plainly rather than claiming as vindication: this client has far less surface to be wrong on than the HTTP/2 one, because §4.6 gave the flow control and the stream bookkeeping to QUIC — and QUIC's own interop suite has been exercising that half all along.
What fuzzing the clients found #
The three clients are fuzzed against arbitrary server bytes —
transport.memory for HTTP/1.1 and HTTP/2, and transport.memory_session for
HTTP/3, which is the seam paying for itself again: an adversarial HTTP/3
response needs no QUIC handshake, no socket and no port. Three real faults in
the first minutes:
- A
204that declares a body. §6.3 rule 1 outranks rule 3 — a 204 has no body "regardless of the header fields present" — soHTTP/1.1 204with aContent-Length, aTransfer-Encodingand five octets is correctly framed as bodiless. But those five octets are still on the wire, and the next request on that connection would be answered by them. The connection is no longer offered for reuse when a bodiless response declared framing. - An HTTP/2 stream that ends with no field section handed back the zero its
status was initialised to — a response that never arrived. §8.1 requires a
message to have a HEADERS frame, so this is now
MalformedResponse. - A leak in the HTTP/3 client, where the arena had an
errdeferand the body list did not, so any error after a DATA frame arrived lost it.
Seeds with interim responses in them found one more, in both multiplexed
clients: a :status of 013. Each checked for three digits and not for a
first digit of at least 1, so a status with no class at all was handed back.
It is MalformedResponse now.
A fourth finding was the property's fault rather than the code's, and is worth
recording for that reason: it asserted a status of 100–599, and the fuzzer
produced HTTP/1.1 804 within seconds. §15 of RFC 9110 defines five classes
and this library deliberately carries a code outside them rather than refusing
it — so the property was wrong, and now checks that such a code reports class
unknown instead.
What is not there yet: a connection pool and cookies. Each is a decision rather than an omission, and the seam they would be built on is the one above.