prefect server in zig

profiling #

This is the in-tree rubric for memory + load analysis. Mirrors the discipline zlay landed on after docs/incident-2026-03-04.md (in ~/tangled.org/zzstoatzz.io/zlay/docs/): instrument first, prove the source with a controlled test, never speculate.

the surface #

/api/metrics (or /metrics, same handler) exposes prometheus text format. Two layers:

  1. Process — read from /proc/self/status on linux; emits zeros with platform="darwin" label on macOS dev:

    • prefect_process_rss_bytes — VmRSS
    • prefect_process_vm_hwm_bytes — VmHWM (peak)
    • prefect_process_rss_anon_bytes — source of truth for total heap (per zlay allocation-audit.md — mallinfo only reports the main glibc arena, RssAnon includes per-thread arenas)
    • prefect_threads_total
  2. Component attribution — counters + gauges from the registry in src/observability.zig. Anything that could grow unboundedly should have an export. Current set:

    • prefect_requests_total, prefect_requests_errors_total
    • prefect_task_workers_* (enqueued, dispatched, requeued, send_failed, subscribers, queue_depth)
    • prefect_task_run_recorder_* (received, matched, applied, stale_dropped)
    • prefect_events_* (subscribers, broadcast_sent, broadcast_dropped)
    • prefect_db_* (query, errors)
    • prefect_automations_* (triggers_fired, actions_executed)
    • prefect_ws_connections_open
  3. glibc mallinfo (linux + libc only) — prefect_malloc_{arena,in_use,free,mmap}_bytes. Useful but incomplete: only the main glibc arena. If RssAnon greatly exceeds malloc_arena_bytes, the leak is in a per-thread arena that mallinfo doesn't see, and the path is wider than malloc — e.g. thread-local arenas, mmap'd buffers, or non-libc-backed allocators.

  4. Build canary — prefect_build_info{git_sha, optimize, zig_version} =

    1. Proves what binary is actually running before you trust the rest. Set git_sha via zig build -Dgit_sha=$(git rev-parse --short HEAD); the justfile build recipes and the docker image (ARG GIT_SHA) do.
  5. Orchestration — two kinds, deliberately:

    • prefect_flow_run_transitions_total{state_type, state_name} — a counter incremented on the one database path every state write takes, so the API process and the services process (late marking, cancellation cleanup, pause expiry) each count what they wrote. Window it with increase(); sum both processes. state_name is a closed set of the standard names, anything custom counts as other, so the label stays bounded however flows name their states. A run's initial state counts when the API creates it; runs the scheduler creates do not, since its idempotent insert cannot tell an insert from an ignore.
    • gauges of state the database owns, read at scrape time in src/db/metrics.zig and served only by the API process: prefect_flow_runs{state_type, state_name} (runs not yet terminal; Late is SCHEDULED/Late), prefect_work_pools{name, type, status, paused}, prefect_deployments{paused}.

    The services process serves its registry on PREFECT_SERVER_METRICS_PORT (default 9000, /metrics, no auth, meant for the cluster network); the helm chart exposes it as the metrics containerPort. Scrape both.

the load driver #

scripts/profile-load exercises a configurable request mix, samples /metrics on a fixed interval, fits a least-squares regression to prefect_process_rss_bytes, and fails if the slope exceeds the threshold.

Default mix (40% deferred task_runs, 20% flow CRUD, 20% events POST, 20% filters) is broad — narrow to one operation when isolating a suspect.

# 5min default
PREFECT_API_URL=http://localhost:4200/api ./scripts/profile-load

# 30min, lower threshold (catch slow leaks)
PROFILE_LOAD_DURATION_S=1800 PROFILE_LOAD_MAX_GROWTH_BPS=10240 ./scripts/profile-load

# isolate task_workers: pin the workload to one op
PROFILE_LOAD_RPS=200 ./scripts/profile-load   # then edit OPS in the script

the rubric for a leak hunt #

  1. Reproduce. just docker-ha-up (or single-replica). just profile-load for the baseline duration with default mix. If slope is flat, the load isn't exercising the leak — adjust the mix.
  2. Attribute. Look at the gauges that grow. A gauge that monotonically increases AND the corresponding *_subscribers.dec site never fires is a leaked-handle smell. A counter alone tells you traffic shape but not whether memory is being held.
  3. Compare against components. Add gauges for any new unbounded structure introduced in the PR before merging. If task_workers_queue_depth grows in lockstep with RSS, that's the leak.
  4. Confirm. Re-run the load with the suspect's hot path disabled (env feature flag, or commented-out enqueue). If RSS goes flat, the leak is confirmed in that path.
  5. Document. Add a new section to docs/allocation-audit.md (mirror of zlay's) describing the file:line that owned the allocation, the lifecycle, and what's keeping it alive. The doc is the institutional memory — every component the load driver covers should have an entry.

anti-patterns (from zlay's incident notes) #

  • don't speculate about causes ("must be the broker") — instrument and prove. Zlay specifically called out "presented theories as facts" as the #1 mistake.
  • don't change multiple variables at once during an investigation. Stack size + optimization level + code change at the same time = no signal.
  • debug build is fine for the investigation. Slower, larger binary, but instrumented. Don't fight ReleaseSafe quirks while also trying to find a leak.
  • verify long enough. Zlay's incident: declared success after seeing a pod running for seconds, not minutes. For us: any growth claim should rest on at least 5 minutes of samples, ideally 30+.

when mallinfo doesn't add up #

If prefect_process_rss_anon_bytes greatly exceeds prefect_malloc_in_use_bytes, you're looking at per-thread arena fragmentation or non-libc allocations:

  • glibc creates per-thread arenas under contention; mallinfo only walks the main arena. Setting MALLOC_ARENA_MAX=2 (env var on the docker container) limits the count and is sometimes the fix for fragmentation without code changes.
  • std.heap.smp_allocator and std.heap.page_allocator both go through mmap → invisible to mallinfo.
  • Stack pages count toward RSS once touched. Lots of threads × moderate per-thread stacks = real RSS that no allocator-level fix can address.

future ambitions (not implemented) #

  • -Duse_gpa=true — switch master allocator to std.heap.DebugAllocator with stack-trace frames. Zlay has this — see src/main.zig gpa block. Requires threading a single allocator through every infra initializer (broker, docket, task_workers, lease_storage) — currently they all reach for std.heap.page_allocator directly. Refactor when we need it.
  • Histograms for request latency. Counter + gauge is enough until we have a latency complaint.
  • OTLP export — logfire-zig is already wired for traces; metrics can follow when we want to consolidate.