- Rust 98%
- TypeScript 1%
- Python <1%
openmw-benchmark #
A performance harness for OpenMW. It drives a scripted scene — or a savegame — against an OpenMW binary, records frame and load telemetry, and tells you whether a build's performance actually changed or whether the difference is noise.
It doesn't build OpenMW and it doesn't patch it. You give it a binary; it gives the binary a temporary home, tells it which scene to load, reads the stats stream it writes back, and does the statistics. That keeps it usable across OpenMW versions and PR branches.
What you need #
- A Rust toolchain. The crate uses edition 2024, so Rust 1.85 or newer.
- An OpenMW build that writes the stats streams. Frame metrics come from the OSG
stats file (
OPENMW_OSG_STATS_FILE), which upstream OpenMW emits. Load metrics come from a load-telemetry stream (OPENMW_LOAD_STATS_FILE); a build without that hook records nothing, and a load run that records nothing fails rather than reporting an empty success.
Build #
cargo build --release
The binary lands in target/release/openmw-benchmark. Tests run with
cargo test, lint with cargo clippy --all-targets -- -D warnings.
Running a benchmark #
openmw-benchmark run --preset balmora --trials 3 --duration 10 --out balmora.json
Each trial launches OpenMW in a throwaway sandbox seeded from a base config,
loads the preset's cell, waits for the warmup frames to pass, then collects
telemetry until the duration elapses (or, with --target-frames, until enough
steady-state frames have been captured). Trials repeat and the per-trial values
feed the statistics. A run with --out writes a JSON report; the same data is
printed to the terminal otherwise.
The binary is found in this order: --bin, then $OPENMW_BIN, then PATH,
then the usual macOS install locations. If nothing turns up you'll get a message
naming all of those.
A savegame can be benchmarked instead of a fresh cell with --save file.omwsave.
--config-dir points at a directory holding the openmw.cfg and settings.cfg
you want the sandbox to start from — useful for pinning settings such as shadow
quality between two runs.
Presets #
A preset is a scene: a cell, a camera position, and a few world settings.
openmw-benchmark list-presets
Four ship with the harness: balmora, ebonheart, pelagiad, and
vivec_arena. --preset also accepts a path to your own JSON file:
{
"id": "balmora",
"name": "Balmora River Vista",
"cell": "Balmora",
"position": [-20650.0, -16700.0, 670.0],
"yaw": 48.7,
"pitch": 4.6,
"time_of_day": 12.0,
"timescale": 0.0,
"disable_ai": true,
"disable_collision": true,
"commands": []
}
Comparing two runs #
openmw-benchmark compare baseline.json candidate.json
openmw-benchmark compare baseline.json candidate.json --markdown > report.md
The comparison tests each headline metric on its per-trial values. Inside a
single suite run the two targets' trials are interleaved, so each baseline trial
is paired with the candidate trial from the same block and a paired t-test is
used, which cancels the shared thermal and cache drift. compare treats its two
inputs as independent runs and uses a Welch t-test, including when they are two
separately executed suite files. The resulting p-values are corrected for
multiple comparisons (Holm-Bonferroni by default, --correction bh for
Benjamini-Hochberg), and each metric is labelled as a significant improvement, a
significant regression, or within noise. Both the raw and adjusted p-value are
printed; the verdict follows the adjusted one, and a line above the table names
the test that produced them.
Reports also summarize measured-trial conditions (median 1-minute load average, worst thermal limits, and median available CPU count). Comparisons warn when the load average differs by more than both 2× and 2.0, a throttling limit below 100% appears on only one side, or available CPU counts differ. These are host-aware diagnostics, not verdict or exit-code changes; a missing or differing host fingerprint makes condition comparisons unverified rather than attributing a cross-machine difference to the build.
For CI:
openmw-benchmark compare baseline.json candidate.json --fail-on-regression
Without the flag compare always exits 0. With it, a significant regression
exits 2 and the offending metrics are written to stderr, so a --markdown
report piped to a file stays clean.
Suites #
A suite runs many scenarios against one or more binaries and reports them together. It's a JSON manifest with shared defaults, a list of targets, and a plan describing which scenes to run:
{
"name": "Shadow matrix",
"defaults": { "trials": 2, "warmup_frames": 80, "target_frames": 150, "duration": 15 },
"targets": [
{ "id": "baseline", "bin": "/path/to/OpenMW-baseline", "label": "0.52 master" },
{ "id": "candidate", "bin": "/path/to/OpenMW-candidate", "label": "shadow PR" }
],
"plan": {
"type": "matrix",
"presets": ["balmora", "pelagiad"],
"configs": [
{ "id": "balanced", "config_dir": "configs/shadows_balanced" },
{ "id": "high", "config_dir": "configs/shadows_high" }
]
}
}
There are three plan types:
- matrix — cross a list of presets with a list of config directories.
- sweep — one scene, one INI key varied across a list of values
(
presets,base_config,section,key,values). - scenarios — an explicit list of scenarios, each with a
presetand optionalconfig_dir,trials,target_frames, and per-inioverrides.
Target IDs and the final expanded scenario IDs must be unique. They key the stored results, so a collision would silently overwrite or skip a scenario; the harness rejects one before any benchmark starts.
A suite also validates its target binaries and presets before the first trial,
so a typo in a later scenario fails immediately. Add --dry-run to print the
expanded plan and an estimated wall clock without running anything.
Run one with:
openmw-benchmark suite suites/shadow_matrix.json --out shadows.json --markdown
Targets are interleaved per trial rather than run in blocks, so drift — thermal,
background load, cache warmup — hits both sides roughly equally. --cooldown
inserts a pause between trials, --resume report.json picks up a suite that
stopped partway, and --continue-on-error keeps going after a scenario fails
(--max-failures bounds it). A suite writes a crash-safety checkpoint next to
--out as it goes, so an interrupted run isn't lost. Ctrl+C (or SIGTERM)
stops the running trial, keeps that checkpoint, and exits 130/143 instead of
leaving OpenMW running in the background.
To compare two suite reports, with or without an embedded A/B target pair:
openmw-benchmark suite-compare baseline_suite.json candidate_suite.json
openmw-benchmark suite-compare baseline_suite.json candidate_suite.json --fail-on-regression
Suite comparisons use the same five headline frame metrics as compare — GPU
and CPU draw, CPU cull, framerate, and frame duration — corrected as one
suite-wide family across comparable scenarios. Each tested metric shows its
raw and adjusted p-value, verdict, and MDE; a significant regression in any
metric can fail --fail-on-regression. Use --out comparison.json to save a
versioned comparison artifact.
Report JSON is written through a same-directory temporary file that is synced
before atomic replacement, so readers never see a partially written report.
The parent directory is not synced, so rename persistence across sudden power
loss is not guaranteed. Replacing a report retains its existing permissions;
new report files are owner-only by default. Suite-report object keys
(scenario_labels, results, and scenario_status, including nested target
maps) are serialized lexicographically; arrays retain their existing scenario
and run order. Saved suite-comparison artifacts carry the report schema version
and are readable by show.
To print a saved report again, without rerunning anything:
openmw-benchmark show baseline_balmora.json
openmw-benchmark show baseline_balmora.json --markdown
show accepts a single-run, suite, or suite-comparison report. run --markdown
and show --markdown emit GitHub/GitLab tables for pasting into an issue or PR.
Load mode #
Everything above measures steady-state rendering. Add --mode load to run or
suite to measure scene and savegame load time instead:
openmw-benchmark run --mode load --preset balmora --trials 5 --warmup-trials 1 \
--phase scene_load_cell --require-phase scene_load_cell --out load.json
In load mode the harness waits until the required phases have appeared and then
ends the trial early, so a load run doesn't sit around for its timeout. Frame
stats aren't required and aren't part of the verdict. --phase picks the phases
the verdict is about, --require-phase fails a trial whose phase never showed up,
and --warmup-trials runs extra leading trials that are recorded but kept out of
the statistics (one by default, to take the edge off the cold cache). If a run
produces no load records at all it fails rather than reporting an empty success.
Load verdicts treat lower as better, and are corrected and gated exactly like the
frame metrics. compare and suite-compare only compare load phases when both
reports are load reports — a framerate run happens to emit some startup load
phases, and those are incidental, not a measurement. When a run declared
--phase, the comparison tests only those phases; incidental phases stay in the
table but out of the correction family and the gate.
Profiling #
On macOS, run can sample the process with /usr/bin/sample:
openmw-benchmark run --preset balmora --profile --flamegraph --open
That writes a raw sample report, a folded-stacks file, a JSON call tree, and an SVG flamegraph. The same rendering is available standalone:
openmw-benchmark flamegraph profile_openmw_balmora.txt --out flame.svg --open
flamegraph takes either a raw sample report or a benchmark JSON, and can
print folded stacks (--folded) or the pruned JSON call tree (--json) for
analysis.
Exit codes #
0— the run or comparison succeeded and no gate tripped.1— the tool failed: bad arguments, unreadable input, a run that couldn't complete.2—--fail-on-regressionfound a significant regression.
2 is deliberately distinct from 1 so a CI job can tell "this build regressed"
from "the harness broke".
Notes #
docs/ has more detail: load-benchmarking.md and
regression-gating.md describe those two features,
roadmap.md tracks what's done and what's next, and
hardening-recommendations.md holds the
open statistical and artifact-hardening tasks.