prefect server in zig

prefect-server: zig vs python — resource & throughput #

Apples-to-apples comparison between this repo's zig prefect-server reimplementation and the canonical python prefect server, on the workloads both can serve. Methodology follows atproto-bench: deterministic workload stream across implementations, variance reported, correctness parity verified, and an explicit "where the gap comes from" breakdown.

All three topologies measured at default configuration. Same workload (40% POST /task_runs/ + 20% POST /flows/ + 20% POST /task_runs/filter), same seed (1337), same conc=16, target 200 rps, 5s warmup + 60s measurement. macOS arm64 (M3 Max). All three topology blocks last refreshed 2026-06-23 on main (602e41f, httpz pinned to first-party http.zig f30aafd) against python Prefect 3.7.5 (uv run --with prefect for local, prefecthq/prefect:3-latest image for container topologies).

TL;DR — local topology (sqlite + in-process broker, both sides) #

Defaults: zig with zqlite.Pool size=5 (matches python's effective default for file sqlite) and the PRAGMA set from python's setup_sqlite(). Python runs prefect server start against a fresh sqlite db.

zig wins python wins
idle/start RSS 34.4 MB vs 312.0 MB (9.1×)
cold start 1.0 s vs ~4 s (~4×)
binary / dist size 6.0 MB vs 177 MB (30×)
p50 latency at conc=16 2.06 vs 7.95 ms (3.86×)
p99 latency at conc=16 14.65 vs 54.82 ms (3.74×)
throughput at conc=16 175.0 vs 42.4 rps (4.13×) — zig error-free; python returns HTTP 503 for most POST /task_runs/ failures plus ReadTimeouts
RSS under load (peak at conc=16) 131.7 MB vs 318.0 MB (2.41×)

Python 3.7.5 note. Python's local throughput came in at ~42 rps on 2026-06-23, back near the 43.8 measured against 3.7.3 on 2026-06-03. Python error mix unchanged: mostly HTTP 503 on POST /task_runs/, plus ReadTimeouts. Also observed: after one 60s bench run, python's sqlalchemy QueuePool (size 5, overflow 10) stays exhausted — a second run against the same process times out on every request while /api/health still returns 200; fresh process required per run.

Sqlite pool sizing note. The local p99 at default size=5 is ~12.5ms. Bisecting the cost: ~2ms comes from multi-conn SQLITE_BUSY retries on this insert-heavy workload (WAL serializes writers anyway), ~3ms is pool acquire/release indirection. Drop to size=3 to reclaim that 2ms (p99 ~10ms) via PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=3 or [server.database.sqlalchemy] pool_size = 3 in prefect.toml — same setting name python uses (their docstring scopes it to postgres; we extend to sqlite). Don't go below 2 if you run background services (event_persister can starve the API path).

TL;DR — single topology (compose.yml / compose.single-python.yml: 1 webapp + services + postgres + redis on both sides) #

Both stacks run identical config: 1 webserver + 1 services container + shared postgres + shared redis. Python uses prefect's official self-hosted setup (prefect_redis.* bridges + docket on redis). Zig uses its native redis broker.

zig wins python wins
p50 latency at conc=16 5.09 vs 9.93 ms (1.95×)
p99 latency at conc=16 95.26 vs 168.39 ms (1.77×)
throughput at conc=16 149.4 vs 20.1 rps (7.43×) — zig error-free; python ReadTimeout failures
combined RSS (webapp + services, peak) 74.9 MB vs 395.7 MB (5.3×)

Single is the topology where the throughput gap widens the most — python sheds throughput vs its own local-sqlite numbers (20 vs 42 rps), apparently the sqlalchemy/asyncpg overhead under postgres+redis dominates at this scale. Zig still holds most of its local rps (149 vs 175) because the postgres path uses pg.zig's own pool and the redis path is the native broker. The current refresh still shows a wider zig p99 tail in postgres/redis topologies than local, so that remains the next perf question to isolate.

TL;DR — HA topology (containerized: 2 webapps + services + postgres + redis on both sides) #

Both stacks run prefect's official HA configuration. Python uses prefect_redis.messaging + prefect_redis.lease_storage + prefect_redis.ordering + docket on redis per the self-hosted docs. Zig uses its native redis broker.

zig wins python wins
dist size per replica image 66 MB vs 1.18 GB (18×)
p50 latency at conc=16 5.67 vs 10.03 ms (1.77×)
p99 latency at conc=16 127.63 vs 134.79 ms (1.06×)
throughput at conc=16 139.3 vs 50.8 rps (2.74×) — zig error-free; python ReadTimeout failures
combined RSS (webapp-a + webapp-b + services, peak) 121.8 MB vs 593.6 MB (4.9×)

HA python is the only topology where python's throughput recovers somewhat (51 rps vs 20 single), because a second webapp absorbs concurrent load. Zig's HA p99 is still around the older ~130ms tail, but python is in the same band and still has ReadTimeout failures. The headline gap remains throughput, RSS, and dist size; latency still favors zig, but the HA p99 margin is narrow.

The zig server wins on every metric in every topology. Earlier revisions of this document reported python winning on latency at low load and on p99 in HA; both were artifacts of un-fair comparison setups (Debug binary, connection: close per response, double-buffered response writes, missing created DESC index on task_run, and python running a "fake HA" that wasn't prefect's recommended pattern). See "history" below.

The harness has explicit local, single/middle, and ha topology modes so postgres/redis and HA runs are not conflated with the sqlite baseline. Both topology defaults are checked in to the repo (compose.yml, compose.ha.yml, compose.ha-python.yml, Dockerfile, Dockerfile.python-ha).

what this measures #

The numbers in this document currently measure the simplest single-node shape:

  • single process (no horizontal scaling, no separate broker container)
  • sqlite for durable state (default for both)
  • in-process broker (memory dialect for zig; python prefect's default in-process messaging)
  • no UI traffic (zig can serve a mounted static UI bundle via PREFECT_UI_STATIC_DIR, but the benchmark hits API routes only)
  • defaults otherwise — same as you get from prefect server start / ./zig-out/bin/prefect-server

scripts/bench-compare also supports the other two deployment shapes we care about:

topology intended user zig shape
local brand-new / ephemeral server one process, sqlite, in-process broker
single / middle small shared server compose.yml: one API container + services, shared postgres + redis
ha company-scale server compose.ha.yml: two API containers + services, shared postgres + redis

For container topologies, implementation RSS is sampled as the sum of the server containers (API replicas + services), not just one API process. Shared postgres/redis RSS is intentionally outside that implementation footprint.

Workload mix mirrors the prefect client's hot path:

op weight what it exercises
POST /task_runs/ (deferred=true) 40% task_run insert + denormalized state row + deferred-task enqueue hook
POST /flows/ 20% upsert by name, generated UUID, simple write
POST /task_runs/filter 20% filtered SELECT, JSON envelope read path

POST /events/in was removed from the mix because zig and python use different transports for event ingestion at this surface (zig is WS-only, python accepts both). It's covered as a parity row below rather than measured.

results #

macOS arm64 (M3 Max), sqlite, in-process broker, fresh DB per server, 5 s warmup + 60 s measurement, conc=16, target 200 rps. Numbers come from a single run; run-to-run variance ~10–15% on rps, ~20–30% on p99.

idle resource footprint #

metric zig python ratio
start RSS (process, benchmark) 33.1 MB 306.2 MB 9.3×
dist size (binary / installed env) 5.2 MB ~177 MB 34×
dependencies 10 direct (zig packages) ~200 transitive (python packages) —
cold start to /api/health 200 1.0 s 3.9 s 3.9×

The python footprint includes the CPython runtime (~50 MB), SQLAlchemy + asyncpg + uvicorn + FastAPI + pydantic v2 + the prefect codebase. The zig binary is statically linked except for libc.

throughput + latency (bench-compare, conc=16) #

concurrency 16 (target 200 rps, 60 s):

metric zig python ratio (zig/py)
successful req/s 170.7 43.8 3.89×
errors over 60 s 0 1338 (1209 × 503, 128 × timeouts, 1 × read error) —
p50 latency (ms) 2.02 8.40 4.16× zig faster
p99 latency (ms) 15.46 29.65 1.92× zig faster
RSS start (MB) 33.1 306.2 0.11× zig
RSS peak (MB) 127.9 310.4 0.41× zig

Per-op at conc=16:

op zig rps / p50 / p99 python rps / p50 / p99
create_task_run 85.7 / 2.04 / 14.99 12.8 / 8.45 / 23.92
create_flow 42.2 / 1.93 / 14.08 15.3 / 7.79 / 31.37
filter_task_runs 42.8 / 2.07 / 17.10 15.7 / 8.98 / 27.42

Python's POST /task_runs/ returns HTTP 503 ("Service Unavailable") for most of its errors at this concurrency — its SQLAlchemy QueuePool(pool_size=5, max_overflow=10) saturates and the surplus comes back as 503s. The remaining failures are httpx-side read timeouts/read errors. Zig serves the same workload with zero errors.

the actual picture (local) #

  1. zig is faster on every operation, at every percentile, with no errors. The p50 lead is ~1.5×; the p99 lead is ~3.6× — the p99 ratio is bigger because python's pool-pressure shows up as long-tail latency before it spills over into 503s.
  2. python collapses to 503s for ~40% of create_task_run requests at conc=16. SQLAlchemy's default pool (5 + 10 overflow = 15 total) saturates and the surplus comes back as 503. The other ops degrade gracefully because they're lower-weight in the mix.
  3. zig wins on RSS at benchmark start (9.3×) and under load (2.43× peak).
  4. No "low-load latency advantage" for python — earlier revisions claimed one, but it was caused by zig-side bugs that we since fixed. See "history" below.

throughput + latency (bench-compare, conc=16, HA topology) #

Both stacks fully containerized: 2 webserver replicas + 1 services container + shared postgres + shared redis. Python configured per prefect's self-hosted docs with prefect_redis.messaging + prefect_redis.cache + prefect_redis.ordering + prefect_redis.lease_storage + docket on redis. Bench harness round-robins requests across the two webapp replicas per side.

metric zig HA python HA ratio (zig/py)
successful req/s 144.0 50.5 2.85×
errors over 60 s 0 128 (ReadTimeout) —
p50 latency (ms) 4.96 9.53 1.92× zig faster
p99 latency (ms) 128.20 135.94 1.06× zig faster
combined RSS start (MB) 43.5 551.8 12.7×
combined RSS peak (MB) 55.5 582.9 10.5×
image size per webserver (MB) 66 1180 18×

Per-op at conc=16, HA:

op zig rps / p50 / p99 python rps / p50 / p99
create_task_run 72.1 / 5.71 / 130.94 24.9 / 10.74 / 135.83
create_flow 35.9 / 4.70 / 126.14 12.7 / 7.81 / 135.05
filter_task_runs 35.9 / 2.94 / 124.11 13.0 / 7.95 / 137.29

the actual picture (HA) #

  1. zig still wins all four metrics under prefect's recommended HA config. Throughput 2.85×, p50 1.92×, p99 1.06×, RSS 10.5×. The narrow p99 lead is honest: this is the topology where both stacks pay the most coordination cost (redis broker + 2-writer postgres contention + docket/cache cross-process), so the gap compresses but doesn't reverse.
  2. The ~130 ms p99 in HA is a topology property, not just a zig regression. Earlier revisions of this document attributed it to "redis broker tail on the deferred-enqueue path," but the diagnosis was too narrow: python HA hits ~136 ms p99 in the same workload, and create_flow (which doesn't touch the broker) also spikes to ~126 ms p99 on zig — so it's broader than the broker. Likely a mix of postgres write contention from two webapp replicas + redis hop + per-replica coordination overhead. Both implementations of this topology pay it.
  3. python errors in HA are still pool exhaustion, but ReadTimeout-shaped instead of 503-shaped — because postgres roundtrip times push the surplus past httpx's 5 s timeout before SQLAlchemy can return the 503.
  4. RSS remains the clearest container gap. Local start RSS is 9.3× smaller; HA combined peak RSS is 10.5× smaller; image size is 18× smaller.

correctness parity #

What each server actually does. The zig column is honest about what's NOT implemented — performance comparisons only make sense for operations both can serve.

capability zig python notes
flow CRUD ✓ ✓ both round-trip name + tags
task_run CRUD + state ✓ ✓ both store state_id + denormalized state cols
state details + message round-trip ✓ ✓ both round-trip state.state_details.{deferred,transition_id,...}
flow_run CRUD + state ✓ ✓ zig matches nebula's denormalized layout
deferred task enqueue (task.delay()) ✓ ✓ both: POST /task_runs/ with state_details.deferred=true triggers per-key queue insert
task_run state events → row update ✓ ✓ both consume prefect.task-run.* events and apply state
deployments CRUD ✓ ✓ zig stores job_variables, schedules, parameters
work_pools + work_queues ✓ ✓ both surface workers, queues, priorities
variables, blocks, artifacts ✓ ✓ zig has all three at REST parity
automations triggers + actions ✓ ✓ zig has its own trigger evaluator + actions executor
concurrency_limits v2 (lease-based) ✓ ✓ zig server implements v2 only
events: POST /events/in (HTTP) — WS only ✓ zig accepts events via WS at /events/in; python accepts HTTP + WS
events: POST /events/filter ✓ ✓ both expose filtered event reads
WS subscriptions: /events/out, /logs/out, /task_runs/subscriptions/scheduled ✓ ✓ all three carry the same protocol shape
migrations CREATE TABLE IF NOT EXISTS at startup alembic zig doesn't preserve schema history; python does (relevant for prod upgrades, not for greenfield perf)
bundled UI — ✓ python's prefect server start serves the SPA on /; zig serves only /api/*
/metrics (prometheus) ✓ — zig exposes /api/metrics with process + per-component counters; python prefect doesn't out of box
/_healthz, /_readyz split only /api/health only /api/health both are single-endpoint; not split for k8s
concurrency_limits v1 (legacy) — ✓ zig deliberately doesn't port v1 — see memory feedback_concurrency_v2_only
OAuth / SSO / multi-workspace — (cloud only) both OSS variants are single-tenant

The performance numbers below apply to the operations marked ✓ in both columns.

where each gap comes from #

This bench shows two separate gaps that move in opposite directions, so they need separate explanations.

RSS gap (zig wins ~9× at benchmark start) #

The 33 MB vs 306 MB benchmark-start footprint is dominated by what each runtime brings in before the first request:

factor zig python prefect approx contribution
runtime static binary, libc only CPython interpreter ~50 MB python baseline
imported codebase only what the binary needs (the prefect-server zig source is ~30 KLOC) full python prefect package (~150 KLOC), pulled into the process at startup ~80 MB
framework stack httpz + native DB/broker clients FastAPI + Starlette + uvicorn + pydantic v2 + SQLAlchemy 2.x + asyncpg/aiosqlite ~50 MB
object model structs, slices, arenas refcounted PyObjects with per-allocation overhead ~20 MB

These add up to the ~300 MB python benchmark-start baseline in the current run. The zig server doesn't not have equivalent code — it just expresses less of it per running process, in part because the feature surface is smaller (see correctness parity) and in part because the languages differ.

latency gap (zig wins, ~4.2× p50 / 1.9× p99 locally) #

At conc=16 in the local topology, zig serves a request in ~2 ms p50; python takes ~8.4 ms p50. Per-op the zig lead is consistent (1.9–2.1 ms across the three ops; python spans 7.8–9.0 ms). The local p99 gap narrows to 15.5 ms vs 29.7 ms because both sides are paying sqlite's serialized-writer cost, while python's pool pressure still spills over into 503s and timeouts.

The python hot path on a successful request is:

uvicorn dispatch → FastAPI routing → pydantic validate
  → SQLAlchemy session.execute → asyncpg/aiosqlite → row mapping
  → pydantic serialize → JSON encode → response

…and the zig hot path is:

wap accept → route table → sqlite mutex → query/exec
  → arena-backed JSON build → keep-alive response

The zig path does measurably less per-request work: no ORM, no pydantic validation, no per-row Python object allocation. The arena tear-down at end-of-request replaces the equivalent of refcount-traversal in python.

saturation gap (zig wins, no contest) #

Python's default SQLAlchemy.QueuePool(pool_size=5, max_overflow=10) means at most 15 concurrent in-flight DB operations. Past that, requests stack up waiting for a connection; sqlite's single writer compounds this. We observe ~1583 × HTTP 503 on POST /task_runs/ at conc=16 — Starlette's "service unavailable" response when the upstream coroutine can't acquire a session in time.

Zig has no equivalent ceiling under the workload tested. The single sqlite-writer lock is the only hard mutual-exclusion point, and it serializes naturally — degrading p99 under load, but no error path.

Both could be tuned (PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE on python; sqlite-mutex granularity on zig). Neither was tuned in this bench. The differences above are what each ships with.

fairness notes #

  • Zig implements a subset of python prefect's feature surface (see correctness parity). Measuring only the operations both support is fair. Measuring the total HTTP route table is not — the zig server doesn't try to be a drop-in for /admin/storage, OAuth, etc.
  • Python prefect is doing more on the hot path even for the operations we measure: pydantic v2 validation per request body, audit-event emission, orchestration policy rule execution (some no-ops in our workload but the policy framework still runs), and SQLAlchemy mapping. We don't subtract those costs from the measurements — the comparison is "what each server actually does when the prefect client POSTs a /task_runs/."
  • The bench driver itself is python (scripts/bench-compare is a uv-script). Driver overhead is constant across both servers and shows up identically in both columns, so the ratio is unaffected.
  • macOS dev environment: numbers from a developer laptop, not a tuned production server. The relative ordering should hold on linux containers; absolute numbers will differ.
  • sqlite single-writer: at concurrency higher than the saturation point, sqlite's writer lock dominates both servers. The zig server hides this less because it doesn't have an async ORM in the way; python's SQLAlchemy queue makes the bottleneck visible as a connection-pool exhaustion error rather than a latency spike.
  • Default configuration only. Neither server was tuned. Python prefect can be configured with a larger SQLAlchemy pool (PREFECT_SERVER_DATABASE_* env vars); the zig server can be moved off the default smp_allocator. Production-tuned numbers would differ from these.

reproducibility #

# zig server (ReleaseFast required — Debug builds will report wrong numbers)
zig build -Doptimize=ReleaseFast
rm -f prefect.db prefect.db-*
PREFECT_SERVER_PORT=4200 ./zig-out/bin/prefect-server &

# python prefect server
rm -f /tmp/py-prefect.db
PREFECT_SERVER_DATABASE_CONNECTION_URL="sqlite+aiosqlite:////tmp/py-prefect.db" \
  uvx prefect server start --host 127.0.0.1 --port 4300 &

# wait for both /api/health to return 200, then:
ZIG_PID=<zig pid> PY_PID=<python child pid, NOT uvx wrapper> \
BENCH_DURATION_S=60 BENCH_CONCURRENCY=16 BENCH_RPS=200 BENCH_WARMUP_S=5 \
  ./scripts/bench-compare

scripts/bench-compare is the driver. Workload mix and per-op weights live at the top of the file; OPS = [...]. Each side consumes the same deterministic operation sequence (BENCH_SEED, default 1337) so differences are not random workload-shape drift. Per-op error signatures are printed alongside the summary so saturation failures are distinguishable from validation failures.

Topology-aware runs:

# local: sqlite + in-process broker, RSS by pid
ZIG_PID=<zig pid> PY_PID=<python child pid> \
  ./scripts/bench-compare --topology local

# single/middle: compose.yml, postgres + redis, RSS by docker containers
docker compose up --build -d webapp services
PY_URL=http://127.0.0.1:4300/api PY_RSS_TARGETS=pid:<python child pid> \
  ./scripts/bench-compare --topology single

# HA: compose.ha.yml (zig) + compose.ha-python.yml (python), both containerized,
#     2 webapps + services + postgres + redis on each side.
docker compose -p prefect-zig-ha -f compose.ha.yml up --build -d
docker compose -p prefect-py-ha  -f compose.ha-python.yml up --build -d
ZIG_RSS_TARGETS="docker:prefect-zig-ha-webapp-a-1,docker:prefect-zig-ha-webapp-b-1,docker:prefect-zig-ha-services-1" \
PY_URLS="http://127.0.0.1:4301/api,http://127.0.0.1:4302/api" \
PY_RSS_TARGETS="docker:prefect-py-ha-webapp-a-1,docker:prefect-py-ha-webapp-b-1,docker:prefect-py-ha-services-1" \
  ./scripts/bench-compare --topology ha

Both topology defaults are checked in. compose.ha-python.yml configures python prefect with prefect_redis.messaging + prefect_redis.cache + prefect_redis.ordering + prefect_redis.lease_storage + docket on redis per prefect's official self-hosted docs — apples-to-apples with the zig HA shape. The docker_rss_kb helper in scripts/bench-compare sums VmRSS across all PIDs in a container (the official prefect image puts tini at PID 1, so single-process reads miss the actual server).

what this doesn't tell you #

  • orchestration policy throughput. The bench hits CRUD-ish paths. State-transition validation (the CoreFlowPolicy rules — RetryFailedFlows, RenameReruns, etc.) is implemented in both servers and runs on every set_state, but the workload mix above only exercises the policy on the deferred-task enqueue path. A workload heavy on state transitions would show a different gap.
  • end-to-end task execution. This is server-only. Worker subprocesses, flow execution, result storage — none are on the measured path.
  • tuned python. We bench python at its defaults (SQLAlchemy.QueuePool(5+10)). Production deployments tune PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE higher. A tuned python run would push more requests through before erroring; latency and RSS deltas would remain.
  • cloud features. Workspaces, RBAC, audit logs, SSO — only python implements them, and only via prefect cloud.

history #

The first revision of this document (committed against a Debug binary with HTTP connection: close per response, double-buffered response writes, and a missing created DESC index on task_run) reported python winning on latency at low concurrency. That conclusion was wrong — all four issues were zig-side bugs, not language fundamentals. Fixing them moved p50 at conc=16 from 147 ms to 4.5 ms.

The fixes:

  1. zig build -Doptimize=ReleaseFast — Debug builds keep bounds checks, undefined-behavior sanitizers, and unoptimized output. The previous revision benched a 20.9 MB Debug binary; ReleaseFast yields a 5.2 MB binary that's the right thing to compare against.
  2. connection: keep-alive default in src/wap/response.zig:sendBody — the previous default forced a TCP handshake per request, which dominated p50 on small bodies.
  3. No more double-buffering in src/wap/response.zig:sendBody — headers go into a small arena-backed buffer, body is written directly. The previous path copied body into a second buffer before writing.
  4. CREATE INDEX ix_task_run__created ON task_run(created DESC) + same on flow_run — added to both src/db/schema/sqlite.zig and src/db/schema/postgres.zig. Without it, every /task_runs/filter was a scan-and-sort over the table.

wap → httpz HTTP+WS swap (this revision) #

Replaced the in-tree src/wap/server.zig HTTP+WS implementation (which abused the WS library's httpFallback to serve plain HTTP on the same port) with the zzstoatzz.io/http.zig fork of karlseguin's http.zig, plus a 17-line addition to upgradeWebsocket for optional subprotocol passthrough (RFC 6455 §1.3 requires the server to echo a chosen subprotocol back; prefect's python events client and worker channel both rely on it). wap.Request / wap.Response now wrap httpz.Request / httpz.Response so handler signatures in src/api/* didn't change.

Delta vs the same workload on the prior wap impl (same release binary, same conc=16, same seed):

topology rps p50 p99
local 169.3 → 170.5 (+0.7%) 2.89 → 2.06 ms (-29%) 12.86 → 12.52 ms (-3%)
single 160.1 → 162.0 (+1.2%) 5.42 → 4.91 ms (-9%) 81.54 → 24.68 ms (-70%)
HA 144.3 → 157.2 (+9%) 5.39 → 5.65 ms (+5%) 131.12 → 29.96 ms (-77%)

Throughput moves are noise-level. The p99 collapse at single + HA is the real signal — wap's per-request alloc dance was the long-tail dominator; httpz's arena-per-request shape replaces it. Also dropped our zzstoatzz/websocket.zig fork (existed only because karlseguin's pre-0.16 master didn't build); upstream master is 0.16-compatible now and pulled in transitively through httpz, so we no longer carry the duplicated websocket package in our build graph.

what's next #

Zig wins in every topology we've measured. Remaining work is feature parity (see ROADMAP.md) and small micro-improvements:

  • release the sqlite mutex before JSON serialization in src/db/backend.zig and src/api/task_runs.zig:filter. Right now lease.query(...) + iterate + write JSON happens under the lock. Pattern would be: query → drain rows into a small slice → release lock → serialize. Useful for read-heavy mixes.
  • HA p99 micro-tail — both stacks land ~110 ms p99 in HA. Could investigate per-replica latency variance and whether redis-broker XADD is dominating the tail on the deferred-enqueue path. Currently a tie at the topology level; small wins still possible.
  • re-run against python with a tuned pool (PREFECT_SERVER_DATABASE_CONNECTION_POOL_SIZE=50) so we can measure python's tuned throughput rather than its default-pool ceiling. The current python loss is partly its own conservative defaults.
  • HA stack (compose.ha.yml) — two zig replicas + shared redis + shared postgres vs two python workers behind a load balancer.
  • state-transition workload — POST /task_runs/{id}/set_state with non-trivial orchestration policies (retries, pause, expire) is the actual hot path in production. The bench mix above is CRUD-shaped and doesn't exercise it.