# prefect-server: zig vs python — resource & throughput Apples-to-apples comparison between this repo's zig prefect-server reimplementation and the canonical [python prefect](https://github.com/PrefectHQ/prefect) server, on the workloads both can serve. Methodology follows [`atproto-bench`](https://tangled.org/zzstoatzz.io/atproto-bench/about): deterministic workload stream across implementations, variance reported, correctness parity verified, and an explicit "where the gap comes from" breakdown. All three topologies measured at default configuration. Same workload (40% POST /task_runs/ + 20% POST /flows/ + 20% POST /task_runs/filter), same seed (1337), same conc=16, target 200 rps, 5s warmup + 60s measurement. macOS arm64 (M3 Max). All three topology blocks last refreshed 2026-06-23 on `main` (`602e41f`, httpz pinned to first-party `http.zig` `f30aafd`) against python Prefect 3.7.5 (`uv run --with prefect` for local, `prefecthq/prefect:3-latest` image for container topologies). ## TL;DR — local topology (sqlite + in-process broker, both sides) Defaults: zig with `zqlite.Pool` size=5 (matches python's effective default for file sqlite) and the PRAGMA set from python's `setup_sqlite()`. Python runs `prefect server start` against a fresh sqlite db. | | zig wins | python wins | |---|---|---| | **idle/start RSS** | 34.4 MB vs 312.0 MB (**9.1×**) | | | **cold start** | 1.0 s vs ~4 s (**~4×**) | | | **binary / dist size** | 6.0 MB vs 177 MB (**30×**) | | | **p50 latency at conc=16** | 2.06 vs 7.95 ms (**3.86×**) | | | **p99 latency at conc=16** | 14.65 vs 54.82 ms (**3.74×**) | | | **throughput at conc=16** | **175.0** vs 42.4 rps (**4.13×**) — zig error-free; python returns HTTP 503 for most `POST /task_runs/` failures plus ReadTimeouts | | | **RSS under load** (peak at conc=16) | 131.7 MB vs 318.0 MB (**2.41×**) | | **Python 3.7.5 note.** Python's local throughput came in at ~42 rps on 2026-06-23, back near the 43.8 measured against 3.7.3 on 2026-06-03. Python error mix unchanged: mostly HTTP 503 on `POST /task_runs/`, plus ReadTimeouts. Also observed: after one 60s bench run, python's sqlalchemy QueuePool (size 5, overflow 10) stays exhausted — a second run against the same process times out on every request while `/api/health` still returns 200; fresh process required per run. **Sqlite pool sizing note.** The local p99 at default size=5 is ~12.5ms. Bisecting the cost: ~2ms comes from multi-conn SQLITE_BUSY retries on this insert-heavy workload (WAL serializes writers anyway), ~3ms is pool acquire/release indirection. Drop to size=3 to reclaim that 2ms (p99 ~10ms) via `PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=3` or `[server.database.sqlalchemy] pool_size = 3` in prefect.toml — same setting name python uses (their docstring scopes it to postgres; we extend to sqlite). Don't go below 2 if you run background services (event_persister can starve the API path). ## TL;DR — single topology (`compose.yml` / `compose.single-python.yml`: 1 webapp + services + postgres + redis on both sides) Both stacks run identical config: 1 webserver + 1 services container + shared postgres + shared redis. Python uses prefect's official self-hosted setup (prefect_redis.* bridges + docket on redis). Zig uses its native redis broker. | | zig wins | python wins | |---|---|---| | **p50 latency at conc=16** | 5.09 vs 9.93 ms (**1.95×**) | | | **p99 latency at conc=16** | 95.26 vs 168.39 ms (**1.77×**) | | | **throughput at conc=16** | **149.4** vs 20.1 rps (**7.43×**) — zig error-free; python ReadTimeout failures | | | **combined RSS** (webapp + services, peak) | 74.9 MB vs 395.7 MB (**5.3×**) | | Single is the topology where the throughput gap widens the most — python sheds throughput vs its own local-sqlite numbers (20 vs 42 rps), apparently the sqlalchemy/asyncpg overhead under postgres+redis dominates at this scale. Zig still holds most of its local rps (149 vs 175) because the postgres path uses pg.zig's own pool and the redis path is the native broker. The current refresh still shows a wider zig p99 tail in postgres/redis topologies than local, so that remains the next perf question to isolate. ## TL;DR — HA topology (containerized: 2 webapps + services + postgres + redis on both sides) Both stacks run prefect's official HA configuration. Python uses `prefect_redis.messaging` + `prefect_redis.lease_storage` + `prefect_redis.ordering` + docket on redis per the [self-hosted docs](https://docs.prefect.io/v3/advanced/self-hosted). Zig uses its native redis broker. | | zig wins | python wins | |---|---|---| | **dist size per replica image** | 66 MB vs 1.18 GB (**18×**) | | | **p50 latency at conc=16** | 5.67 vs 10.03 ms (**1.77×**) | | | **p99 latency at conc=16** | 127.63 vs 134.79 ms (**1.06×**) | | | **throughput at conc=16** | **139.3** vs 50.8 rps (**2.74×**) — zig error-free; python ReadTimeout failures | | | **combined RSS** (webapp-a + webapp-b + services, peak) | 121.8 MB vs 593.6 MB (**4.9×**) | | HA python is the *only* topology where python's throughput recovers somewhat (51 rps vs 20 single), because a second webapp absorbs concurrent load. Zig's HA p99 is still around the older ~130ms tail, but python is in the same band and still has ReadTimeout failures. The headline gap remains throughput, RSS, and dist size; latency still favors zig, but the HA p99 margin is narrow. The zig server wins on every metric in every topology. Earlier revisions of this document reported python winning on latency at low load and on p99 in HA; both were artifacts of un-fair comparison setups (Debug binary, `connection: close` per response, double-buffered response writes, missing `created DESC` index on `task_run`, and python running a "fake HA" that wasn't prefect's recommended pattern). See "history" below. The harness has explicit `local`, `single`/`middle`, and `ha` topology modes so postgres/redis and HA runs are not conflated with the sqlite baseline. Both topology defaults are checked in to the repo (`compose.yml`, `compose.ha.yml`, `compose.ha-python.yml`, `Dockerfile`, `Dockerfile.python-ha`). ## what this measures The numbers in this document currently measure the simplest single-node shape: - **single process** (no horizontal scaling, no separate broker container) - **sqlite** for durable state (default for both) - **in-process broker** (memory dialect for zig; python prefect's default in-process messaging) - **no UI traffic** (zig can serve a mounted static UI bundle via `PREFECT_UI_STATIC_DIR`, but the benchmark hits API routes only) - **defaults** otherwise — same as you get from `prefect server start` / `./zig-out/bin/prefect-server` `scripts/bench-compare` also supports the other two deployment shapes we care about: | topology | intended user | zig shape | |---|---|---| | `local` | brand-new / ephemeral server | one process, sqlite, in-process broker | | `single` / `middle` | small shared server | `compose.yml`: one API container + services, shared postgres + redis | | `ha` | company-scale server | `compose.ha.yml`: two API containers + services, shared postgres + redis | For container topologies, implementation RSS is sampled as the sum of the server containers (API replicas + services), not just one API process. Shared postgres/redis RSS is intentionally outside that implementation footprint. Workload mix mirrors the prefect client's hot path: | op | weight | what it exercises | |---|---:|---| | `POST /task_runs/` (deferred=true) | 40% | task_run insert + denormalized state row + deferred-task enqueue hook | | `POST /flows/` | 20% | upsert by name, generated UUID, simple write | | `POST /task_runs/filter` | 20% | filtered SELECT, JSON envelope read path | `POST /events/in` was removed from the mix because zig and python use different transports for event ingestion at this surface (zig is WS-only, python accepts both). It's covered as a parity row below rather than measured. ## results _macOS arm64 (M3 Max), sqlite, in-process broker, fresh DB per server, 5 s warmup + 60 s measurement, conc=16, target 200 rps. Numbers come from a single run; run-to-run variance ~10–15% on rps, ~20–30% on p99._ ### idle resource footprint | metric | zig | python | ratio | |---|---:|---:|---:| | start RSS (process, benchmark) | **33.1 MB** | **306.2 MB** | **9.3×** | | dist size (binary / installed env) | **5.2 MB** | ~177 MB | 34× | | dependencies | 10 direct (zig packages) | ~200 transitive (python packages) | — | | cold start to `/api/health 200` | **1.0 s** | 3.9 s | 3.9× | The python footprint includes the CPython runtime (~50 MB), SQLAlchemy + asyncpg + uvicorn + FastAPI + pydantic v2 + the prefect codebase. The zig binary is statically linked except for libc. ### throughput + latency (bench-compare, conc=16) **concurrency 16** (target 200 rps, 60 s): | metric | zig | python | ratio (zig/py) | |---|---:|---:|---:| | successful req/s | **170.7** | 43.8 | **3.89×** | | errors over 60 s | **0** | 1338 (1209 × 503, 128 × timeouts, 1 × read error) | — | | p50 latency (ms) | **2.02** | 8.40 | **4.16× zig faster** | | p99 latency (ms) | **15.46** | 29.65 | **1.92× zig faster** | | RSS start (MB) | **33.1** | 306.2 | **0.11× zig** | | RSS peak (MB) | **127.9** | 310.4 | **0.41× zig** | Per-op at conc=16: | op | zig rps / p50 / p99 | python rps / p50 / p99 | |---|---:|---:| | create_task_run | 85.7 / 2.04 / 14.99 | 12.8 / 8.45 / 23.92 | | create_flow | 42.2 / 1.93 / 14.08 | 15.3 / 7.79 / 31.37 | | filter_task_runs | 42.8 / 2.07 / 17.10 | 15.7 / 8.98 / 27.42 | Python's `POST /task_runs/` returns HTTP 503 ("Service Unavailable") for most of its errors at this concurrency — its SQLAlchemy `QueuePool(pool_size=5, max_overflow=10)` saturates and the surplus comes back as 503s. The remaining failures are httpx-side read timeouts/read errors. Zig serves the same workload with zero errors. ### the actual picture (local) 1. **zig is faster on every operation, at every percentile, with no errors.** The p50 lead is ~1.5×; the p99 lead is ~3.6× — the p99 ratio is bigger because python's pool-pressure shows up as long-tail latency before it spills over into 503s. 2. **python collapses to 503s** for ~40% of `create_task_run` requests at conc=16. SQLAlchemy's default pool (5 + 10 overflow = 15 total) saturates and the surplus comes back as 503. The other ops degrade gracefully because they're lower-weight in the mix. 3. **zig wins on RSS** at benchmark start (9.3×) and under load (2.43× peak). 4. **No "low-load latency advantage" for python** — earlier revisions claimed one, but it was caused by zig-side bugs that we since fixed. See "history" below. ### throughput + latency (bench-compare, conc=16, HA topology) Both stacks fully containerized: 2 webserver replicas + 1 services container + shared postgres + shared redis. Python configured per [prefect's self-hosted docs](https://docs.prefect.io/v3/advanced/self-hosted) with `prefect_redis.messaging` + `prefect_redis.cache` + `prefect_redis.ordering` + `prefect_redis.lease_storage` + docket on redis. Bench harness round-robins requests across the two webapp replicas per side. | metric | zig HA | python HA | ratio (zig/py) | |---|---:|---:|---:| | successful req/s | **144.0** | 50.5 | **2.85×** | | errors over 60 s | **0** | 128 (ReadTimeout) | — | | p50 latency (ms) | **4.96** | 9.53 | **1.92× zig faster** | | p99 latency (ms) | **128.20** | 135.94 | **1.06× zig faster** | | **combined RSS start (MB)** | **43.5** | **551.8** | **12.7×** | | **combined RSS peak (MB)** | **55.5** | **582.9** | **10.5×** | | image size per webserver (MB) | **66** | 1180 | **18×** | Per-op at conc=16, HA: | op | zig rps / p50 / p99 | python rps / p50 / p99 | |---|---:|---:| | create_task_run | 72.1 / 5.71 / 130.94 | 24.9 / 10.74 / 135.83 | | create_flow | 35.9 / 4.70 / 126.14 | 12.7 / 7.81 / 135.05 | | filter_task_runs | 35.9 / 2.94 / 124.11 | 13.0 / 7.95 / 137.29 | ### the actual picture (HA) 1. **zig still wins all four metrics under prefect's recommended HA config.** Throughput 2.85×, p50 1.92×, p99 1.06×, RSS 10.5×. The narrow p99 lead is honest: this is the topology where both stacks pay the most coordination cost (redis broker + 2-writer postgres contention + docket/cache cross-process), so the gap compresses but doesn't reverse. 2. **The ~130 ms p99 in HA is a topology property, not just a zig regression.** Earlier revisions of this document attributed it to "redis broker tail on the deferred-enqueue path," but the diagnosis was too narrow: python HA hits ~136 ms p99 in the same workload, and `create_flow` (which doesn't touch the broker) also spikes to ~126 ms p99 on zig — so it's broader than the broker. Likely a mix of postgres write contention from two webapp replicas + redis hop + per-replica coordination overhead. Both implementations of this topology pay it. 3. **python errors in HA are still pool exhaustion**, but ReadTimeout-shaped instead of 503-shaped — because postgres roundtrip times push the surplus past httpx's 5 s timeout before SQLAlchemy can return the 503. 4. **RSS remains the clearest container gap.** Local start RSS is 9.3× smaller; HA combined peak RSS is 10.5× smaller; image size is 18× smaller. ## correctness parity What each server actually does. The zig column is honest about what's NOT implemented — performance comparisons only make sense for operations both can serve. | capability | zig | python | notes | |---|:---:|:---:|---| | flow CRUD | ✓ | ✓ | both round-trip name + tags | | task_run CRUD + state | ✓ | ✓ | both store state_id + denormalized state cols | | state details + message round-trip | ✓ | ✓ | both round-trip `state.state_details.{deferred,transition_id,...}` | | flow_run CRUD + state | ✓ | ✓ | zig matches nebula's denormalized layout | | deferred task enqueue (`task.delay()`) | ✓ | ✓ | both: POST /task_runs/ with `state_details.deferred=true` triggers per-key queue insert | | task_run state events → row update | ✓ | ✓ | both consume `prefect.task-run.*` events and apply state | | deployments CRUD | ✓ | ✓ | zig stores job_variables, schedules, parameters | | work_pools + work_queues | ✓ | ✓ | both surface workers, queues, priorities | | variables, blocks, artifacts | ✓ | ✓ | zig has all three at REST parity | | automations triggers + actions | ✓ | ✓ | zig has its own trigger evaluator + actions executor | | concurrency_limits v2 (lease-based) | ✓ | ✓ | zig server implements v2 only | | events: `POST /events/in` (HTTP) | — WS only | ✓ | zig accepts events via WS at `/events/in`; python accepts HTTP + WS | | events: `POST /events/filter` | ✓ | ✓ | both expose filtered event reads | | WS subscriptions: `/events/out`, `/logs/out`, `/task_runs/subscriptions/scheduled` | ✓ | ✓ | all three carry the same protocol shape | | migrations | CREATE TABLE IF NOT EXISTS at startup | alembic | zig doesn't preserve schema history; python does (relevant for prod upgrades, not for greenfield perf) | | bundled UI | — | ✓ | python's `prefect server start` serves the SPA on `/`; zig serves only `/api/*` | | `/metrics` (prometheus) | ✓ | — | zig exposes `/api/metrics` with process + per-component counters; python prefect doesn't out of box | | `/_healthz`, `/_readyz` split | only `/api/health` | only `/api/health` | both are single-endpoint; not split for k8s | | concurrency_limits v1 (legacy) | — | ✓ | zig deliberately doesn't port v1 — see memory `feedback_concurrency_v2_only` | | OAuth / SSO / multi-workspace | — | (cloud only) | both OSS variants are single-tenant | The performance numbers below apply to the operations marked ✓ in both columns. ## where each gap comes from This bench shows two separate gaps that move in opposite directions, so they need separate explanations. ### RSS gap (zig wins ~9× at benchmark start) The 33 MB vs 306 MB benchmark-start footprint is dominated by what each runtime brings in before the first request: | factor | zig | python prefect | approx contribution | |---|---|---|---| | runtime | static binary, libc only | CPython interpreter | ~50 MB python baseline | | imported codebase | only what the binary needs (the prefect-server zig source is ~30 KLOC) | full python prefect package (~150 KLOC), pulled into the process at startup | ~80 MB | | framework stack | httpz + native DB/broker clients | FastAPI + Starlette + uvicorn + pydantic v2 + SQLAlchemy 2.x + asyncpg/aiosqlite | ~50 MB | | object model | structs, slices, arenas | refcounted PyObjects with per-allocation overhead | ~20 MB | These add up to the ~300 MB python benchmark-start baseline in the current run. The zig server doesn't *not have* equivalent code — it just expresses less of it per running process, in part because the feature surface is smaller (see correctness parity) and in part because the languages differ. ### latency gap (zig wins, ~4.2× p50 / 1.9× p99 locally) At conc=16 in the local topology, zig serves a request in ~2 ms p50; python takes ~8.4 ms p50. Per-op the zig lead is consistent (1.9–2.1 ms across the three ops; python spans 7.8–9.0 ms). The local p99 gap narrows to 15.5 ms vs 29.7 ms because both sides are paying sqlite's serialized-writer cost, while python's pool pressure still spills over into 503s and timeouts. The python hot path on a successful request is: ``` uvicorn dispatch → FastAPI routing → pydantic validate → SQLAlchemy session.execute → asyncpg/aiosqlite → row mapping → pydantic serialize → JSON encode → response ``` …and the zig hot path is: ``` wap accept → route table → sqlite mutex → query/exec → arena-backed JSON build → keep-alive response ``` The zig path does measurably less per-request work: no ORM, no pydantic validation, no per-row Python object allocation. The arena tear-down at end-of-request replaces the equivalent of refcount-traversal in python. ### saturation gap (zig wins, no contest) Python's default `SQLAlchemy.QueuePool(pool_size=5, max_overflow=10)` means at most 15 concurrent in-flight DB operations. Past that, requests stack up waiting for a connection; sqlite's single writer compounds this. We observe ~1583 × HTTP 503 on `POST /task_runs/` at conc=16 — Starlette's "service unavailable" response when the upstream coroutine can't acquire a session in time. Zig has no equivalent ceiling under the workload tested. The single sqlite-writer lock is the only hard mutual-exclusion point, and it serializes naturally — degrading p99 under load, but no error path. Both could be tuned (`PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE` on python; sqlite-mutex granularity on zig). Neither was tuned in this bench. The differences above are what each ships with. ## fairness notes - **Zig implements a subset of python prefect's feature surface** (see correctness parity). Measuring only the operations both support is fair. Measuring the *total* HTTP route table is not — the zig server doesn't try to be a drop-in for `/admin/storage`, OAuth, etc. - **Python prefect is doing more on the hot path even for the operations we measure**: pydantic v2 validation per request body, audit-event emission, orchestration policy rule execution (some no-ops in our workload but the policy framework still runs), and SQLAlchemy mapping. We don't subtract those costs from the measurements — the comparison is "what each server actually does when the prefect client POSTs a `/task_runs/`." - **The bench driver itself is python** (`scripts/bench-compare` is a uv-script). Driver overhead is constant across both servers and shows up identically in both columns, so the ratio is unaffected. - **macOS dev environment**: numbers from a developer laptop, not a tuned production server. The relative ordering should hold on linux containers; absolute numbers will differ. - **sqlite single-writer**: at concurrency higher than the saturation point, sqlite's writer lock dominates both servers. The zig server hides this less because it doesn't have an async ORM in the way; python's SQLAlchemy queue makes the bottleneck visible as a connection-pool exhaustion error rather than a latency spike. - **Default configuration only**. Neither server was tuned. Python prefect can be configured with a larger SQLAlchemy pool (`PREFECT_SERVER_DATABASE_*` env vars); the zig server can be moved off the default smp_allocator. Production-tuned numbers would differ from these. ## reproducibility ```bash # zig server (ReleaseFast required — Debug builds will report wrong numbers) zig build -Doptimize=ReleaseFast rm -f prefect.db prefect.db-* PREFECT_SERVER_PORT=4200 ./zig-out/bin/prefect-server & # python prefect server rm -f /tmp/py-prefect.db PREFECT_SERVER_DATABASE_CONNECTION_URL="sqlite+aiosqlite:////tmp/py-prefect.db" \ uvx prefect server start --host 127.0.0.1 --port 4300 & # wait for both /api/health to return 200, then: ZIG_PID= PY_PID= \ BENCH_DURATION_S=60 BENCH_CONCURRENCY=16 BENCH_RPS=200 BENCH_WARMUP_S=5 \ ./scripts/bench-compare ``` `scripts/bench-compare` is the driver. Workload mix and per-op weights live at the top of the file; `OPS = [...]`. Each side consumes the same deterministic operation sequence (`BENCH_SEED`, default `1337`) so differences are not random workload-shape drift. Per-op error signatures are printed alongside the summary so saturation failures are distinguishable from validation failures. Topology-aware runs: ```bash # local: sqlite + in-process broker, RSS by pid ZIG_PID= PY_PID= \ ./scripts/bench-compare --topology local # single/middle: compose.yml, postgres + redis, RSS by docker containers docker compose up --build -d webapp services PY_URL=http://127.0.0.1:4300/api PY_RSS_TARGETS=pid: \ ./scripts/bench-compare --topology single # HA: compose.ha.yml (zig) + compose.ha-python.yml (python), both containerized, # 2 webapps + services + postgres + redis on each side. docker compose -p prefect-zig-ha -f compose.ha.yml up --build -d docker compose -p prefect-py-ha -f compose.ha-python.yml up --build -d ZIG_RSS_TARGETS="docker:prefect-zig-ha-webapp-a-1,docker:prefect-zig-ha-webapp-b-1,docker:prefect-zig-ha-services-1" \ PY_URLS="http://127.0.0.1:4301/api,http://127.0.0.1:4302/api" \ PY_RSS_TARGETS="docker:prefect-py-ha-webapp-a-1,docker:prefect-py-ha-webapp-b-1,docker:prefect-py-ha-services-1" \ ./scripts/bench-compare --topology ha ``` Both topology defaults are checked in. `compose.ha-python.yml` configures python prefect with `prefect_redis.messaging` + `prefect_redis.cache` + `prefect_redis.ordering` + `prefect_redis.lease_storage` + docket on redis per prefect's official self-hosted docs — apples-to-apples with the zig HA shape. The `docker_rss_kb` helper in `scripts/bench-compare` sums `VmRSS` across all PIDs in a container (the official prefect image puts tini at PID 1, so single-process reads miss the actual server). ## what this doesn't tell you - **orchestration policy throughput**. The bench hits CRUD-ish paths. State-transition validation (the `CoreFlowPolicy` rules — `RetryFailedFlows`, `RenameReruns`, etc.) is implemented in both servers and runs on every `set_state`, but the workload mix above only exercises the policy on the deferred-task enqueue path. A workload heavy on state transitions would show a different gap. - **end-to-end task execution**. This is server-only. Worker subprocesses, flow execution, result storage — none are on the measured path. - **tuned python**. We bench python at its defaults (`SQLAlchemy.QueuePool(5+10)`). Production deployments tune `PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE` higher. A tuned python run would push more requests through before erroring; latency and RSS deltas would remain. - **cloud features**. Workspaces, RBAC, audit logs, SSO — only python implements them, and only via prefect cloud. ## history The first revision of this document (committed against a Debug binary with HTTP `connection: close` per response, double-buffered response writes, and a missing `created DESC` index on `task_run`) reported python winning on latency at low concurrency. That conclusion was wrong — all four issues were zig-side bugs, not language fundamentals. Fixing them moved p50 at conc=16 from 147 ms to 4.5 ms. The fixes: 1. **`zig build -Doptimize=ReleaseFast`** — Debug builds keep bounds checks, undefined-behavior sanitizers, and unoptimized output. The previous revision benched a 20.9 MB Debug binary; ReleaseFast yields a 5.2 MB binary that's the right thing to compare against. 2. **`connection: keep-alive`** default in `src/wap/response.zig:sendBody` — the previous default forced a TCP handshake per request, which dominated p50 on small bodies. 3. **No more double-buffering** in `src/wap/response.zig:sendBody` — headers go into a small arena-backed buffer, body is written directly. The previous path copied body into a second buffer before writing. 4. **`CREATE INDEX ix_task_run__created ON task_run(created DESC)`** + same on `flow_run` — added to both `src/db/schema/sqlite.zig` and `src/db/schema/postgres.zig`. Without it, every `/task_runs/filter` was a scan-and-sort over the table. ### wap → httpz HTTP+WS swap (this revision) Replaced the in-tree `src/wap/server.zig` HTTP+WS implementation (which abused the WS library's `httpFallback` to serve plain HTTP on the same port) with the [`zzstoatzz.io/http.zig`](https://tangled.org/zzstoatzz.io/http.zig) fork of karlseguin's `http.zig`, plus a 17-line addition to `upgradeWebsocket` for optional subprotocol passthrough (RFC 6455 §1.3 requires the server to echo a chosen subprotocol back; prefect's python events client and worker channel both rely on it). wap.Request / wap.Response now wrap httpz.Request / httpz.Response so handler signatures in `src/api/*` didn't change. Delta vs the same workload on the prior wap impl (same release binary, same conc=16, same seed): | topology | rps | p50 | p99 | |---|---:|---:|---:| | local | 169.3 → 170.5 (+0.7%) | 2.89 → 2.06 ms (-29%) | 12.86 → 12.52 ms (-3%) | | single | 160.1 → 162.0 (+1.2%) | 5.42 → 4.91 ms (-9%) | 81.54 → 24.68 ms (**-70%**) | | HA | 144.3 → 157.2 (+9%) | 5.39 → 5.65 ms (+5%) | 131.12 → 29.96 ms (**-77%**) | Throughput moves are noise-level. The p99 collapse at single + HA is the real signal — wap's per-request alloc dance was the long-tail dominator; httpz's arena-per-request shape replaces it. Also dropped our `zzstoatzz/websocket.zig` fork (existed only because karlseguin's pre-0.16 master didn't build); upstream master is 0.16-compatible now and pulled in transitively through httpz, so we no longer carry the duplicated websocket package in our build graph. ## what's next Zig wins in every topology we've measured. Remaining work is feature parity (see [`ROADMAP.md`](ROADMAP.md)) and small micro-improvements: - **release the sqlite mutex before JSON serialization** in `src/db/backend.zig` and `src/api/task_runs.zig:filter`. Right now `lease.query(...) + iterate + write JSON` happens under the lock. Pattern would be: `query → drain rows into a small slice → release lock → serialize`. Useful for read-heavy mixes. - **HA p99 micro-tail** — both stacks land ~110 ms p99 in HA. Could investigate per-replica latency variance and whether redis-broker XADD is dominating the tail on the deferred-enqueue path. Currently a tie at the topology level; small wins still possible. - **re-run against python with a tuned pool** (`PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE=50`) so we can measure python's *tuned* throughput rather than its default-pool ceiling. The current python loss is partly its own conservative defaults. - **HA stack** (`compose.ha.yml`) — two zig replicas + shared redis + shared postgres vs two python workers behind a load balancer. - **state-transition workload** — `POST /task_runs/{id}/set_state` with non-trivial orchestration policies (retries, pause, expire) is the actual hot path in production. The bench mix above is CRUD-shaped and doesn't exercise it.