profiling #
This is the in-tree rubric for memory + load analysis. Mirrors the
discipline zlay landed on after docs/incident-2026-03-04.md (in
~/tangled.org/zzstoatzz.io/zlay/docs/): instrument first, prove the
source with a controlled test, never speculate.
the surface #
/api/metrics (or /metrics, same handler) exposes prometheus text format.
Two layers:
-
Process — read from
/proc/self/statuson linux; emits zeros withplatform="darwin"label on macOS dev:prefect_process_rss_bytes— VmRSSprefect_process_vm_hwm_bytes— VmHWM (peak)prefect_process_rss_anon_bytes— source of truth for total heap (per zlay allocation-audit.md — mallinfo only reports the main glibc arena, RssAnon includes per-thread arenas)prefect_threads_total
-
Component attribution — counters + gauges from the registry in
src/observability.zig. Anything that could grow unboundedly should have an export. Current set:prefect_requests_total,prefect_requests_errors_totalprefect_task_workers_*(enqueued, dispatched, requeued, send_failed, subscribers, queue_depth)prefect_task_run_recorder_*(received, matched, applied, stale_dropped)prefect_events_*(subscribers, broadcast_sent, broadcast_dropped)prefect_db_*(query, errors)prefect_automations_*(triggers_fired, actions_executed)prefect_ws_connections_open
-
glibc mallinfo (linux + libc only) —
prefect_malloc_{arena,in_use,free,mmap}_bytes. Useful but incomplete: only the main glibc arena. IfRssAnongreatly exceedsmalloc_arena_bytes, the leak is in a per-thread arena that mallinfo doesn't see, and the path is wider than malloc — e.g. thread-local arenas, mmap'd buffers, or non-libc-backed allocators. -
Build canary —
prefect_build_info{git_sha, optimize, zig_version}=- Proves what binary is actually running before you trust the rest.
Set
git_shaviazig build -Dgit_sha=$(git rev-parse --short HEAD); the justfile build recipes and the docker image (ARG GIT_SHA) do.
- Proves what binary is actually running before you trust the rest.
Set
-
Orchestration — two kinds, deliberately:
prefect_flow_run_transitions_total{state_type, state_name}— a counter incremented on the one database path every state write takes, so the API process and the services process (late marking, cancellation cleanup, pause expiry) each count what they wrote. Window it withincrease(); sum both processes.state_nameis a closed set of the standard names, anything custom counts asother, so the label stays bounded however flows name their states. A run's initial state counts when the API creates it; runs the scheduler creates do not, since its idempotent insert cannot tell an insert from an ignore.- gauges of state the database owns, read at scrape time in
src/db/metrics.zigand served only by the API process:prefect_flow_runs{state_type, state_name}(runs not yet terminal; Late isSCHEDULED/Late),prefect_work_pools{name, type, status, paused},prefect_deployments{paused}.
The services process serves its registry on
PREFECT_SERVER_METRICS_PORT(default 9000,/metrics, no auth, meant for the cluster network); the helm chart exposes it as themetricscontainerPort. Scrape both.
the load driver #
scripts/profile-load exercises a configurable request mix, samples
/metrics on a fixed interval, fits a least-squares regression to
prefect_process_rss_bytes, and fails if the slope exceeds the threshold.
Default mix (40% deferred task_runs, 20% flow CRUD, 20% events POST, 20% filters) is broad — narrow to one operation when isolating a suspect.
# 5min default
PREFECT_API_URL=http://localhost:4200/api ./scripts/profile-load
# 30min, lower threshold (catch slow leaks)
PROFILE_LOAD_DURATION_S=1800 PROFILE_LOAD_MAX_GROWTH_BPS=10240 ./scripts/profile-load
# isolate task_workers: pin the workload to one op
PROFILE_LOAD_RPS=200 ./scripts/profile-load # then edit OPS in the script
the rubric for a leak hunt #
- Reproduce.
just docker-ha-up(or single-replica).just profile-loadfor the baseline duration with default mix. If slope is flat, the load isn't exercising the leak — adjust the mix. - Attribute. Look at the gauges that grow. A gauge that monotonically
increases AND the corresponding
*_subscribers.decsite never fires is a leaked-handle smell. A counter alone tells you traffic shape but not whether memory is being held. - Compare against components. Add gauges for any new unbounded
structure introduced in the PR before merging. If
task_workers_queue_depthgrows in lockstep with RSS, that's the leak. - Confirm. Re-run the load with the suspect's hot path disabled (env feature flag, or commented-out enqueue). If RSS goes flat, the leak is confirmed in that path.
- Document. Add a new section to
docs/allocation-audit.md(mirror of zlay's) describing the file:line that owned the allocation, the lifecycle, and what's keeping it alive. The doc is the institutional memory — every component the load driver covers should have an entry.
anti-patterns (from zlay's incident notes) #
- don't speculate about causes ("must be the broker") — instrument and prove. Zlay specifically called out "presented theories as facts" as the #1 mistake.
- don't change multiple variables at once during an investigation. Stack size + optimization level + code change at the same time = no signal.
- debug build is fine for the investigation. Slower, larger binary, but instrumented. Don't fight ReleaseSafe quirks while also trying to find a leak.
- verify long enough. Zlay's incident: declared success after seeing a pod running for seconds, not minutes. For us: any growth claim should rest on at least 5 minutes of samples, ideally 30+.
when mallinfo doesn't add up #
If prefect_process_rss_anon_bytes greatly exceeds
prefect_malloc_in_use_bytes, you're looking at per-thread arena
fragmentation or non-libc allocations:
- glibc creates per-thread arenas under contention; mallinfo only walks
the main arena. Setting
MALLOC_ARENA_MAX=2(env var on the docker container) limits the count and is sometimes the fix for fragmentation without code changes. std.heap.smp_allocatorandstd.heap.page_allocatorboth go through mmap → invisible to mallinfo.- Stack pages count toward RSS once touched. Lots of threads × moderate per-thread stacks = real RSS that no allocator-level fix can address.
future ambitions (not implemented) #
-Duse_gpa=true— switch master allocator tostd.heap.DebugAllocatorwith stack-trace frames. Zlay has this — seesrc/main.ziggpa block. Requires threading a single allocator through every infra initializer (broker, docket, task_workers, lease_storage) — currently they all reach forstd.heap.page_allocatordirectly. Refactor when we need it.- Histograms for request latency. Counter + gauge is enough until we have a latency complaint.
- OTLP export — logfire-zig is already wired for traces; metrics can follow when we want to consolidate.