# profiling This is the in-tree rubric for memory + load analysis. Mirrors the discipline zlay landed on after `docs/incident-2026-03-04.md` (in `~/tangled.org/zzstoatzz.io/zlay/docs/`): **instrument first, prove the source with a controlled test, never speculate.** ## the surface `/api/metrics` (or `/metrics`, same handler) exposes prometheus text format. Two layers: 1. **Process** — read from `/proc/self/status` on linux; emits zeros with `platform="darwin"` label on macOS dev: - `prefect_process_rss_bytes` — VmRSS - `prefect_process_vm_hwm_bytes` — VmHWM (peak) - `prefect_process_rss_anon_bytes` — **source of truth for total heap** (per zlay allocation-audit.md — mallinfo only reports the main glibc arena, RssAnon includes per-thread arenas) - `prefect_threads_total` 2. **Component attribution** — counters + gauges from the registry in `src/observability.zig`. Anything that could grow unboundedly should have an export. Current set: - `prefect_requests_total`, `prefect_requests_errors_total` - `prefect_task_workers_*` (enqueued, dispatched, requeued, send_failed, subscribers, queue_depth) - `prefect_task_run_recorder_*` (received, matched, applied, stale_dropped) - `prefect_events_*` (subscribers, broadcast_sent, broadcast_dropped) - `prefect_db_*` (query, errors) - `prefect_automations_*` (triggers_fired, actions_executed) - `prefect_ws_connections_open` 3. **glibc mallinfo** (linux + libc only) — `prefect_malloc_{arena,in_use,free,mmap}_bytes`. Useful but **incomplete**: only the main glibc arena. If `RssAnon` greatly exceeds `malloc_arena_bytes`, the leak is in a per-thread arena that mallinfo doesn't see, and the path is wider than malloc — e.g. thread-local arenas, mmap'd buffers, or non-libc-backed allocators. 4. **Build canary** — `prefect_build_info{git_sha, optimize, zig_version}` = 1. Proves what binary is actually running before you trust the rest. Set `git_sha` via `zig build -Dgit_sha=$(git rev-parse --short HEAD)`; the justfile build recipes and the docker image (`ARG GIT_SHA`) do. 5. **Orchestration** — two kinds, deliberately: - `prefect_flow_run_transitions_total{state_type, state_name}` — a counter incremented on the one database path every state write takes, so the API process and the services process (late marking, cancellation cleanup, pause expiry) each count what they wrote. Window it with `increase()`; sum both processes. `state_name` is a closed set of the standard names, anything custom counts as `other`, so the label stays bounded however flows name their states. A run's initial state counts when the API creates it; runs the scheduler creates do not, since its idempotent insert cannot tell an insert from an ignore. - gauges of state the database owns, read at scrape time in `src/db/metrics.zig` and served only by the API process: `prefect_flow_runs{state_type, state_name}` (runs not yet terminal; Late is `SCHEDULED`/`Late`), `prefect_work_pools{name, type, status, paused}`, `prefect_deployments{paused}`. The services process serves its registry on `PREFECT_SERVER_METRICS_PORT` (default 9000, `/metrics`, no auth, meant for the cluster network); the helm chart exposes it as the `metrics` containerPort. Scrape both. ## the load driver `scripts/profile-load` exercises a configurable request mix, samples `/metrics` on a fixed interval, fits a least-squares regression to `prefect_process_rss_bytes`, and fails if the slope exceeds the threshold. Default mix (40% deferred task_runs, 20% flow CRUD, 20% events POST, 20% filters) is broad — narrow to one operation when isolating a suspect. ```bash # 5min default PREFECT_API_URL=http://localhost:4200/api ./scripts/profile-load # 30min, lower threshold (catch slow leaks) PROFILE_LOAD_DURATION_S=1800 PROFILE_LOAD_MAX_GROWTH_BPS=10240 ./scripts/profile-load # isolate task_workers: pin the workload to one op PROFILE_LOAD_RPS=200 ./scripts/profile-load # then edit OPS in the script ``` ## the rubric for a leak hunt 1. **Reproduce.** `just docker-ha-up` (or single-replica). `just profile-load` for the baseline duration with default mix. If slope is flat, the load isn't exercising the leak — adjust the mix. 2. **Attribute.** Look at the gauges that grow. A gauge that monotonically increases AND the corresponding `*_subscribers.dec` site never fires is a leaked-handle smell. A counter alone tells you traffic shape but not whether memory is being held. 3. **Compare against components.** Add gauges for any *new* unbounded structure introduced in the PR before merging. If `task_workers_queue_depth` grows in lockstep with RSS, that's the leak. 4. **Confirm.** Re-run the load with the suspect's hot path disabled (env feature flag, or commented-out enqueue). If RSS goes flat, the leak is confirmed in that path. 5. **Document.** Add a new section to `docs/allocation-audit.md` (mirror of zlay's) describing the file:line that owned the allocation, the lifecycle, and what's keeping it alive. The doc is the institutional memory — every component the load driver covers should have an entry. ## anti-patterns (from zlay's incident notes) - **don't speculate** about causes ("must be the broker") — instrument and prove. Zlay specifically called out "presented theories as facts" as the #1 mistake. - **don't change multiple variables** at once during an investigation. Stack size + optimization level + code change at the same time = no signal. - **debug build is fine** for the investigation. Slower, larger binary, but instrumented. Don't fight ReleaseSafe quirks while also trying to find a leak. - **verify long enough.** Zlay's incident: declared success after seeing a pod running for seconds, not minutes. For us: any growth claim should rest on at least 5 minutes of samples, ideally 30+. ## when mallinfo doesn't add up If `prefect_process_rss_anon_bytes` greatly exceeds `prefect_malloc_in_use_bytes`, you're looking at per-thread arena fragmentation or non-libc allocations: - glibc creates per-thread arenas under contention; mallinfo only walks the main arena. Setting `MALLOC_ARENA_MAX=2` (env var on the docker container) limits the count and is sometimes the fix for fragmentation without code changes. - `std.heap.smp_allocator` and `std.heap.page_allocator` both go through mmap → invisible to mallinfo. - Stack pages count toward RSS once touched. Lots of threads × moderate per-thread stacks = real RSS that no allocator-level fix can address. ## future ambitions (not implemented) - **`-Duse_gpa=true`** — switch master allocator to `std.heap.DebugAllocator` with stack-trace frames. Zlay has this — see `src/main.zig` gpa block. Requires threading a single allocator through every infra initializer (broker, docket, task_workers, lease_storage) — currently they all reach for `std.heap.page_allocator` directly. Refactor when we need it. - **Histograms** for request latency. Counter + gauge is enough until we have a latency complaint. - **OTLP export** — logfire-zig is already wired for traces; metrics can follow when we want to consolidate.