external benchmarking harness for OpenMW
README.md

openmw-benchmark #

A performance harness for OpenMW. It drives a scripted scene — or a savegame — against an OpenMW binary, records frame and load telemetry, and tells you whether a build's performance actually changed or whether the difference is noise.

It doesn't build OpenMW and it doesn't patch it. You give it a binary; it gives the binary a temporary home, tells it which scene to load, reads the stats stream it writes back, and does the statistics. That keeps it usable across OpenMW versions and PR branches.

What you need #

  • A Rust toolchain. The crate uses edition 2024, so Rust 1.85 or newer.
  • An OpenMW build that writes the stats streams. Frame metrics come from the OSG stats file (OPENMW_OSG_STATS_FILE), which upstream OpenMW emits. Load metrics come from a load-telemetry stream (OPENMW_LOAD_STATS_FILE); a build without that hook records nothing, and a load run that records nothing fails rather than reporting an empty success.

Build #

cargo build --release

The binary lands in target/release/openmw-benchmark. Tests run with cargo test, lint with cargo clippy --all-targets -- -D warnings.

Running a benchmark #

openmw-benchmark run --preset balmora --trials 3 --duration 10 --out balmora.json

Each trial launches OpenMW in a throwaway sandbox seeded from a base config, loads the preset's cell, waits for the warmup frames to pass, then collects telemetry until the duration elapses (or, with --target-frames, until enough steady-state frames have been captured). Trials repeat and the per-trial values feed the statistics. A run with --out writes a JSON report; the same data is printed to the terminal otherwise.

The binary is found in this order: --bin, then $OPENMW_BIN, then PATH, then the usual macOS install locations. If nothing turns up you'll get a message naming all of those.

A savegame can be benchmarked instead of a fresh cell with --save file.omwsave. --config-dir points at a directory holding the openmw.cfg and settings.cfg you want the sandbox to start from — useful for pinning settings such as shadow quality between two runs.

Presets #

A preset is a scene: a cell, a camera position, and a few world settings.

openmw-benchmark list-presets

Four ship with the harness: balmora, ebonheart, pelagiad, and vivec_arena. --preset also accepts a path to your own JSON file:

{
  "id": "balmora",
  "name": "Balmora River Vista",
  "cell": "Balmora",
  "position": [-20650.0, -16700.0, 670.0],
  "yaw": 48.7,
  "pitch": 4.6,
  "time_of_day": 12.0,
  "timescale": 0.0,
  "disable_ai": true,
  "disable_collision": true,
  "commands": []
}

Comparing two runs #

openmw-benchmark compare baseline.json candidate.json
openmw-benchmark compare baseline.json candidate.json --markdown > report.md

The comparison tests each headline metric on its per-trial values. Inside a single suite run the two targets' trials are interleaved, so each baseline trial is paired with the candidate trial from the same block and a paired t-test is used, which cancels the shared thermal and cache drift. compare treats its two inputs as independent runs and uses a Welch t-test, including when they are two separately executed suite files. The resulting p-values are corrected for multiple comparisons (Holm-Bonferroni by default, --correction bh for Benjamini-Hochberg), and each metric is labelled as a significant improvement, a significant regression, or within noise. Both the raw and adjusted p-value are printed; the verdict follows the adjusted one, and a line above the table names the test that produced them.

Reports also summarize measured-trial conditions (median 1-minute load average, worst thermal limits, and median available CPU count). Comparisons warn when the load average differs by more than both 2× and 2.0, a throttling limit below 100% appears on only one side, or available CPU counts differ. These are host-aware diagnostics, not verdict or exit-code changes; a missing or differing host fingerprint makes condition comparisons unverified rather than attributing a cross-machine difference to the build.

For CI:

openmw-benchmark compare baseline.json candidate.json --fail-on-regression

Without the flag compare always exits 0. With it, a significant regression exits 2 and the offending metrics are written to stderr, so a --markdown report piped to a file stays clean.

Suites #

A suite runs many scenarios against one or more binaries and reports them together. It's a JSON manifest with shared defaults, a list of targets, and a plan describing which scenes to run:

{
  "name": "Shadow matrix",
  "defaults": { "trials": 2, "warmup_frames": 80, "target_frames": 150, "duration": 15 },
  "targets": [
    { "id": "baseline",  "bin": "/path/to/OpenMW-baseline",  "label": "0.52 master" },
    { "id": "candidate", "bin": "/path/to/OpenMW-candidate", "label": "shadow PR" }
  ],
  "plan": {
    "type": "matrix",
    "presets": ["balmora", "pelagiad"],
    "configs": [
      { "id": "balanced", "config_dir": "configs/shadows_balanced" },
      { "id": "high",     "config_dir": "configs/shadows_high" }
    ]
  }
}

There are three plan types:

  • matrix — cross a list of presets with a list of config directories.
  • sweep — one scene, one INI key varied across a list of values (presets, base_config, section, key, values).
  • scenarios — an explicit list of scenarios, each with a preset and optional config_dir, trials, target_frames, and per-ini overrides.

Target IDs and the final expanded scenario IDs must be unique. They key the stored results, so a collision would silently overwrite or skip a scenario; the harness rejects one before any benchmark starts.

A suite also validates its target binaries and presets before the first trial, so a typo in a later scenario fails immediately. Add --dry-run to print the expanded plan and an estimated wall clock without running anything.

Run one with:

openmw-benchmark suite suites/shadow_matrix.json --out shadows.json --markdown

Targets are interleaved per trial rather than run in blocks, so drift — thermal, background load, cache warmup — hits both sides roughly equally. --cooldown inserts a pause between trials, --resume report.json picks up a suite that stopped partway, and --continue-on-error keeps going after a scenario fails (--max-failures bounds it). A suite writes a crash-safety checkpoint next to --out as it goes, so an interrupted run isn't lost. Ctrl+C (or SIGTERM) stops the running trial, keeps that checkpoint, and exits 130/143 instead of leaving OpenMW running in the background.

To compare two suite reports, with or without an embedded A/B target pair:

openmw-benchmark suite-compare baseline_suite.json candidate_suite.json
openmw-benchmark suite-compare baseline_suite.json candidate_suite.json --fail-on-regression

Suite comparisons use the same five headline frame metrics as compare — GPU and CPU draw, CPU cull, framerate, and frame duration — corrected as one suite-wide family across comparable scenarios. Each tested metric shows its raw and adjusted p-value, verdict, and MDE; a significant regression in any metric can fail --fail-on-regression. Use --out comparison.json to save a versioned comparison artifact.

Report JSON is written through a same-directory temporary file that is synced before atomic replacement, so readers never see a partially written report. The parent directory is not synced, so rename persistence across sudden power loss is not guaranteed. Replacing a report retains its existing permissions; new report files are owner-only by default. Suite-report object keys (scenario_labels, results, and scenario_status, including nested target maps) are serialized lexicographically; arrays retain their existing scenario and run order. Saved suite-comparison artifacts carry the report schema version and are readable by show.

To print a saved report again, without rerunning anything:

openmw-benchmark show baseline_balmora.json
openmw-benchmark show baseline_balmora.json --markdown

show accepts a single-run, suite, or suite-comparison report. run --markdown and show --markdown emit GitHub/GitLab tables for pasting into an issue or PR.

Load mode #

Everything above measures steady-state rendering. Add --mode load to run or suite to measure scene and savegame load time instead:

openmw-benchmark run --mode load --preset balmora --trials 5 --warmup-trials 1 \
    --phase scene_load_cell --require-phase scene_load_cell --out load.json

In load mode the harness waits until the required phases have appeared and then ends the trial early, so a load run doesn't sit around for its timeout. Frame stats aren't required and aren't part of the verdict. --phase picks the phases the verdict is about, --require-phase fails a trial whose phase never showed up, and --warmup-trials runs extra leading trials that are recorded but kept out of the statistics (one by default, to take the edge off the cold cache). If a run produces no load records at all it fails rather than reporting an empty success.

Load verdicts treat lower as better, and are corrected and gated exactly like the frame metrics. compare and suite-compare only compare load phases when both reports are load reports — a framerate run happens to emit some startup load phases, and those are incidental, not a measurement. When a run declared --phase, the comparison tests only those phases; incidental phases stay in the table but out of the correction family and the gate.

Profiling #

On macOS, run can sample the process with /usr/bin/sample:

openmw-benchmark run --preset balmora --profile --flamegraph --open

That writes a raw sample report, a folded-stacks file, a JSON call tree, and an SVG flamegraph. The same rendering is available standalone:

openmw-benchmark flamegraph profile_openmw_balmora.txt --out flame.svg --open

flamegraph takes either a raw sample report or a benchmark JSON, and can print folded stacks (--folded) or the pruned JSON call tree (--json) for analysis.

Exit codes #

  • 0 — the run or comparison succeeded and no gate tripped.
  • 1 — the tool failed: bad arguments, unreadable input, a run that couldn't complete.
  • 2 — --fail-on-regression found a significant regression.

2 is deliberately distinct from 1 so a CI job can tell "this build regressed" from "the harness broke".

Notes #

docs/ has more detail: load-benchmarking.md and regression-gating.md describe those two features, roadmap.md tracks what's done and what's next, and hardening-recommendations.md holds the open statistical and artifact-hardening tasks.