Performance checks #
swift test runs headless smoke workloads and fixed debug CPU ceilings. It launches
no app and requests neither Accessibility nor Screen Recording. Hosted CI prints
that it is smoke-only. A smoke report cannot pass a performance comparison.
Regression measurements #
make bench-record first requires a successful quiet-check, then deliberately
writes Benchmarks/baselines/<host>-<model>-<cores>.json. The host is LocalHostName
plus eight SHA-256 digits of the platform UUID; the UUID itself is never recorded.
Review and commit a baseline deliberately. None has been recorded on this busy Mac.
make bench compares the current host's baseline. Schema v5 rejects a different
host, macOS build, Swift version, build configuration, counter source, case set,
sampling configuration, or checksum. An OS/compiler upgrade requires re-recording,
not an automatic migration. Override paths with BENCH_BASELINE and BENCH_OUTPUT.
Synchronous cases gate on median retired instructions per operation: +3% fails. Per-thread counters are probed across known work before measurement; zero or stuck counters crash locally. Public process counters are an explicitly announced fallback. Intel without counters must use a quiet-verified CPU-only comparison. Hosted guests with unavailable counters remain smoke-only and never record zeros.
Three warm-up batches precede 15 batches of at least 10 ms. CPU and wall times, cycles, proof counters, and retained malloc blocks/bytes are also recorded. CPU/time regressions above 25%, and absolute release CPU budgets, gate only when both runs have verified quiet conditions. Wall/CPU/instruction p99 needs at least 1,000 latency samples. Expensive soak/hung-app workloads omit p99 instead of labelling a small sample as p99. Async work does not claim per-thread instruction counts. Smoke runs use 30 soak ticks instead of 10,000 and yield between workload groups, so headless checks do not monopolize the main actor during unrelated UI tests. Actual tick counts remain in the proof metrics; full release runs keep 10,000.
Malloc-zone statistics measure retained allocations, not total malloc calls.
They cannot establish an exact mallocs/tick gate. Layer-identity proof counters
measure newly reachable layers separately; Instruments is needed for total churn.
Calibration is diagnostic only; BENCH_CALIBRATE=1 now fails explicitly.
make bench-ab BASE=main builds the merge base and working tree before five
interleaved A/B rounds. Both use the same v5 instrumentation overlay and deterministic
hashing. The default common-core scope compares the existing 44 cases, so an older
base need not have new testing hooks. Scripts/bench-ab.sh main --all-cases includes
all compatible cases. The summary reports instruction ratios and a deterministic
10,000-resample CPU bootstrap interval. Instructions above +3%, or a CPU interval's
lower bound above 1.05, fail. Reporting-only cases never gate.
Workload catalog #
build/bench-catalog.json contains every new requested benchmark's name, budget,
execution kind, status, and reporting-only reason. Executable cases produce actual
measurements in build/bench-results.json. Cases awaiting another package's code
fix are reporting-only. Missing hooks or live-only probes are explicitly listed as
unsupported; they do not emit fake successful measurements.
Fixtures cover 10/50/200 windows, multiple workspaces/displays, drag ticks, mixed turns, snapshot subscribers, full motion ticks, retargeting, layers, config reload, input handlers, and sanitized recorded window-list decoding. Synthetic backend method counts are not described as measured AX calls or actor hops.
TabView/CanvasAndCardRaster/12 constructs an offscreen canvas and directly draws
12 cards into a 2x bitmap. It is reporting-only: it excludes app snapshot/layout,
input delivery, panel ordering and presentation. It is warmed, not a cold open;
all cards use the runner PID, whose icon may be unavailable. The 100 latency
samples do not produce a p99. It cannot establish the separate
TabView/OpenToFirstFrame/12 target of under 50 ms wall time.
The new executable case changes baseline coverage; re-record on a quiet host.
The open-to-first-frame case needs a harness-owned live fixture with 12 windows.
Record a monotonic start timestamp in the Tab View branch of NativeAppRuntime's
hotkey callback, before openTabView reads the snapshot and minimized windows.
Correlate it with a presentation-confirmed frame containing the expected cards,
using the same clock. Panel ordering, a transaction commit, or a display tick alone
does not prove presentation. Alternatively, an opt-in screen capture can detect
those pixels and report the first observed capture timestamp and sampling delay;
that is an observation bound, not an exact display presentation timestamp. Keep
Screen Recording permission out of the headless lane.
Live measurement targets #
Quit your running Parcel first. Only the harness-owned build/Parcel.app needs its
Accessibility grant; the runner needs neither Accessibility nor Screen Recording.
make smoothness: six fixture scenarios, 20 trials each, raw write/vsync JSON, input latency, settling, starvation, gaps, jank, overshoot, and writes/refresh. Quiet readings permit live gates; otherwise results are reporting-only.make idle: five-second settle, 60-second CPU/instruction/interrupt/package-idle deltas across settled, focused/settings-closed, and config-reloaded states.PARCEL_IDLE_SPACE=1adds a native Space roundtrip. Failure captures sample/logs.make idle-report PID=...: optional sudo power diagnostics for a running app.make soak: 30 fixture windows, 2,000 commands, footprint every 100 commands; final <40 MB and last-ten-sample slope <8 KB/command.make leaks: explicitly inspectable debug bundle, stack logging, soak handshake, and leak graph. Only the harness-owned PID is inspected. Release cannot request the debug inspection entitlement.make trace MODE=smoothness SECONDS=30: attach to an existing app, never launch it.make trace-e2e SCENARIO=swap-2-windows: record a harness-owned scenario and retain its test verdict. Export withScripts/trace.sh export <recording.trace>.
Custom combined Instruments templates need genuine GUI exports. Missing templates
fail clearly; TRACE_OPTIONS=--installed-template explicitly selects a
non-equivalent installed preset. See Instruments/README.md for the missing lanes.
Emit-only trace sites use recorder isEnabled; transaction bookkeeping remains
behind PARCEL_TRACE=1. Input records the event timestamp numerically because the
supported signpost API cannot backdate an interval.
Host safety #
make quiet-check writes every condition to build/bench-quiet.json: load <1.5,
AC power, no thermal warning, no sampled process >10% CPU, Low Power Mode off.
It exits 3 with reasons when busy. Compare tolerates that only with counters;
BENCH_STRICT_QUIET=1 makes quietness mandatory.
make bench-remote HOST=<authorized-host> TASK=bench requires an explicit host.
Nothing here selects or contacts another Mac automatically. Remote runs wait for
quietness, keep the host awake, copy artifacts back, and leave recorded baselines
unstaged. Scripts/bench-host-setup.sh <host> is dry-run unless --apply is explicit.
No nightly job is installed. Benchmarks/hosts.md lists the preparation checklist.
Benchmarks/idle-waits.json documents every current bounded wait; unit tests reject
new unreviewed waits, repeating watchdogs, strict timers, and stale allowances.
It is a static guard, not proof that the live idle budget passes.