Evaluation integration validation #
Validated on 2026-09-13. This is integration evidence, not a published model-quality
comparison or qualification of the concurrently changing serving recipe.
This document covers the current pinned integration in eval/scripts and
eval/configs. Concurrent historical adapters under eval/direct are separate
work and were not audited here. Their dependency metadata was preserved.
Passed #
- Exact Prime Envs and Verifiers source checkouts; locked evaluator installation
(
verifiers 0.3.2.dev86);setup --checkconfirms source, dependency and native runtime-export identities without changing them. A second, newly created temporary virtual environment installed all 112 platform packages from the same lock and passed version/import/native CLI help checks; the normal environment was not replaced. - All 17 profile entries passed native configuration validation: the smoke
ran end to end, and final quick/full/agentic/agentic-full wrapper dry runs
passed with the actual local dataset routing. All core entries use
Docker with the tool-free
nullharness; optional entries usebash. - All pinned source datasets downloaded and hash-checked. Native offline loaders were exercised with fresh derived caches: AIME24/25/26, I3 Logic, LiveCodeBench, GraphWalks. MRCR generation-pinned source files are present and verified.
- Sandbox image built from pinned Python/uv images and native uv script locks.
Both fixed upstream helper images and
sandbox --checkpass. The image also passed native locked script-environment checks with network disabled. No serving container was stopped, replaced or reconfigured by evaluation. - Authenticated AIME24 3 × 1 evaluation completed through Prime Envs against
http://127.0.0.1:18020/v1, modelqwen3.8-27b. Three native episodes haveok=true; two calls reached the 8192-token limit. Truncation is retained in the official records, not treated as an infrastructure success score.
The successful local run is ignored:
eval/runs/smoke.1YiECYH9/aime24/aime24--qwen3.8-27b--null--152f0430/.
Its parent contains provenance. The resolved JSON confirms the intended model,
client URL/key-variable, three tasks, one rollout, C1, temperature 0.6, top-p
0.95, top-k 20, 8192-token budget, thinking enabled, null, local Docker,
/workspace, 2 CPUs, 4 GB, restricted networking, static one-worker serving
and push=false.
The exact sandbox image for that run was
sha256:aedce3045f6e36ec790d545666ca680cf3b1695fa8e08fccc3baf28b377d56dd.
The model was the already-running 64K baseline, not the newly edited candidate.
Its observed stock image was
sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9,
with a mounted Nix source overlay. Its serving-recipe commit was not established.
The recorded checkout was dirty. Do not use this smoke run for a release claim
or matched recipe comparison.
2026-09-25: MMLU-Pro and the broad lane profiles #
mmlu-pro 0.1.0(editable, pinned Prime Envs checkout) was added topyproject.toml;uv lock --offlineadded only that package (dependency:verifiers, already locked) anduv sync --locked --offlineinstalled it.setup --checkpasses.datasets.lockpinsTIGER-Lab/MMLU-Proatb189ec765aa7ed75c8acfea42df31fdae71f97be, the revision the taskset hardcodes:README.md(configs metadata),data/test-00000-of-00001.parquetanddata/validation-00000-of-00001.parquet(the default config's two splits).data mmlu-prodownloaded anddata --check mmlu-proverified them.- The native offline loader (relative
TIGER-Lab/MMLU-Prolink in a fresh working directory, fresh Arrow cache, HF offline) selected the 20 seed-0 shuffled zero-shot tasks, andrun tiny --dry-runandrun broad --dry-runpassed. For every lane taskset the lane's offline selection replay produced exactly the resolved config of the native--dry-run; the AIME25 selection and resolved config are byte-identical to the historical v1 lane's. No model endpoint was used; no task or scorer was changed.
2026-10-04: audit of the current pins #
setup --check,data --check all(all seven Hub snapshots and both MRCR files) andrun PROFILE --dry-runfor all eight profiles (21 entries) pass. Every resolved config namesqwen3.8-27bathttp://127.0.0.1:18020/v1withQWEN_API_KEY, greedy sampling and the profile's budget.direct/setup --checkanddirect/run smoke|quick|full --dry-runpass.- The serving-container default
qwen-inference-inference-1was stale (a never-started old container).runnow records the container named byQWEN_SERVING_CONTAINERonly if it publishes theconfigs/local.tomlendpoint, and otherwise the one running container that does. sandbox --checkfailed ("Sandbox build inputs changed"), so every non-dryrunstopped before its first native command. The recorded imagesha256:facda9cb…was built from the pre-MMLU-Propyproject.toml/uv.lock; only those two recipe inputs changed, and neither enters the build context.eval/scripts/sandbox(innix develop) rebuilt and recordedsha256:5962c155…. Againstfacda9cb…, all 10,400 non-.pycfiles have equal content except 35 uv HTTP-cache.httpentries (fetch metadata).sandbox --checkpasses. Later lane runs record the new image ID.
2026-10-04: hash-locked build tools #
- Before:
uv syncbuilt the editable Verifiers and Prime Envs packages in isolated build environments.build-constraint-dependenciespinned hatchling, hatch-vcs and setuptools-scm by version only. uv fetched them and their dependencies without hashes. uv 0.12.1 has no hashed build-constraint form. - Now:
pyproject.toml(anddirect/mrcr,direct/graphwalks) lists the build tools in a defaultbuilddependency group and setsno-build-isolation.uv.lockrecords them with hashes (editables 0.6, hatchling 1.32.0, hatch-vcs 0.5.0, setuptools-scm 9.2.2, setuptools 84.0.0, packaging 26.3, pathspec 1.1.1, pluggy 1.6.0, tomlkit 0.15.1, trove-classifiers 2026.6.1.19). Every locked hash matches the PyPI JSON API. uv installs these first and then builds the editables in the project environment, so no build fetches an unlocked file. - Verifiers has a dynamic version.
[[tool.uv.dependency-metadata]]gives uv its static metadata, so the lock does not need a build. uv does not compare it with the build.setupanddirect/setupnow build the metadata with the locked hatchling and fail on any difference (shown with a changed version and with a removed requirement). uv lock --offlinechangeduv.lockonly by these additions, the dependency-metadata entry and the Verifiers lock metadata. No runtime package version changed. A new virtual environment synced offline with--locked; uv reported "Prepared 9 packages without build isolation".setup --check,direct/setup --check,run tiny --dry-runanddirect/run smoke --dry-runpass.pyproject.tomlanduv.lockare sandbox recipe inputs, sosandbox --checkfailed after this change.eval/scripts/sandboxre-recorded the image assha256:d6a4517c…. Its RootFS layer list is identical tosha256:5962c155…; neither file enters the image build context.sandbox --checkpasses.direct/setupnow runs uv with--directory: uv 0.12.1 reports the dynamic-version lock as stale with--projectfrom another directory. It also setsGIT_LFS_SKIP_SMUDGE=1and uses the Nixpython3.12.
2026-10-04: Dependabot advisories #
- urllib3 2.7.0 -> 2.8.0 (GHSA-vxq7-64xx-v4gw, GHSA-8988-9cw3-xx77,
GHSA-gh4c-6fx4-qh6g) and PyJWT 2.14.0 -> 2.15.0 (GHSA-42vr-xj54-vc7v) in
uv.lock,direct/graphwalks/uv.lockanddirect/mrcr/uv.lock.uv lock --upgrade-package urllib3==2.8.0 --upgrade-package pyjwt==2.15.0(uv 0.12.1,nix develop) changed only these two packages and their hashes. Nopyproject.tomlchanged. Thebuildgroup is unchanged and stays hashed. - datasets stays 4.6.1 in
direct/. GHSA-379c-qx7v-6h59 is fixed in 5.0.1, but the pinned Verifiers 0.1.15.dev17 declaresdatasets>=3.0.0,<4.7.0.direct/setup --checkrunsuv pip checkand fails on that conflict. The advisory is a path traversal throughfile_namemetadata in folder-based builders (imagefolder, audiofolder, videofolder). The direct runtime does not use them:- A search of the pinned Verifiers tree (
977e3fc4, bothverifiersandverifiers-graphwalkscheckouts),graphwalks.py,mrcr_v2.pyandeval/direct/{run,setup,inspect_dataset.py,prepare_transport.py,configs}finds noimagefolder,audiofolder,videofolder,FolderBasedBuilder,file_nameorsave_to_disk. - The one
push_to_hubisverifiers/utils/save_utils.py:884. It runs only ifsave_to_hf_hubis true (utils/eval_utils.py:1134,envs/environment.py:1102). All 11direct/configs/*/*.tomlsetsave_to_hf_hub = false(line 11). - GraphWalks:
graphwalks.py:188callsload_dataset("openai/graphwalks").runlinks that name to the snapshotf338bb26…. It holds onlyREADME.mdand two parquet files, sha25654036036…and53787943…indatasets.lock. The smoke dry run recordedbuilder_name: parquet, 1150 rows. - MRCR:
mrcr_v2.pydoes not importdatasets. It reads the two CSV files withcsv.DictReader(mrcr_v2.py:113-114).datasets.lockpins their GCS generation, size and sha256 (c6be39bc…,fe3b726c…).runchecks them withscripts/data --checkbefore it loads them. The download atmrcr_v2.py:99-102runs only for a missing file.
- A search of the pinned Verifiers tree (
setupanddirect/setup(sync mode) installed the new versions.setup --check,direct/setup --check,run tiny --dry-runanddirect/run smoke --dry-runpass.eval/scripts/sandboxre-recorded the image assha256:93ac158c…. Its RootFS layer list is identical tosha256:d6a4517c….sandbox --checkpasses.
2026-10-13: Dependabot advisories #
- multidict 6.8.0 -> 6.9.1 (GHSA-54p9-h82j-f925, reference leak in
items-view union and subtraction) in
uv.lock,direct/graphwalks/uv.lockanddirect/mrcr/uv.lock.uv lock --upgrade-package multidict==6.9.1(uv 0.12.1) changed only multidict and its hashes. Nopyproject.tomlchanged. - fsspec stays 2026.2.0 in
direct/. GHSA-27vj-qcqg-25rc is fixed in 2026.6.0, but datasets 4.6.1 declaresfsspec[http]<=2026.2.0and the pinned Verifiers 0.1.15.dev17 declaresdatasets<4.7.0. The advisory is Jinja2 template injection inReferenceFileSystem(protocolreference, Kerchunk JSON). The direct runtime does not open it:- A search of
direct/.sources(both Verifiers checkouts, Prime Envs),direct/{mrcr,graphwalks,run,configs,inspect_dataset.py,prepare_transport.py}and the installed packages of both direct environments findsReferenceFileSystemandreference://only in fsspec itself (registry.py,implementations/reference.py). - GraphWalks loads a local parquet snapshot; MRCR reads two CSV files with
csv.DictReader. Both are hash-pinned indatasets.lock.
- A search of
setup,direct/setup(sync mode),setup --check,direct/setup --check,data --check all,run tiny --dry-runanddirect/run smoke --dry-runpass.scripts/sandboxre-recorded the image assha256:2ff2ed54…;sandbox --checkpasses.
Failures found and fixed during integration #
A raw HF snapshot is not enough for a fresh offline repository-name lookup.
Native local dataset paths fix AIME; I3 needs its logic/ subdirectory and the
local parquet builder's default config. GraphWalks uses native local relative
path discovery from its recorded per-run working directory. No tasks, questions,
answers or scorers were changed.
The current AIME TaskData requires a network policy. Verifiers correctly rejects its execution on the subprocess runtime. Docker preserves that policy. Earlier failed attempts remain ignored and were not silently retried or counted as model quality results. A missing key file fails before any model request.
Repository checks #
ShellCheck with -x and shell syntax checks pass for the evaluation scripts.
No throughput or capacity figures were remeasured during this refactor.
Nix flake validation passed on a source-only snapshot that excluded ignored
caches, datasets and traces and included the concurrent untracked runtime files.
Git-backed development-shell evaluation also passed. Do not use path:. on a
checkout containing private ignored runs.
Full repository checks were run read-only. The inspected tree reported 164 Ruff diagnostics, 9 ty diagnostics, and one formatting failure in since-removed model-preparation code. These concern retained/concurrent runtime and preparation code, not newly added evaluation Python (there is none). Strict rules were not disabled and authenticated/vendor inputs were not rewritten for lint. No repository test suite is configured; no tests were added or represented as passing. Runtime files were changing concurrently, so these counts identify the inspected state rather than a promise about later edits.
Not claimed #
Quick/full quality suites and agentic model runs have not been executed. Native configuration and dataset-loader checks are not substitutes for those runs. No 64K/240K capacity or model-quality parity claim follows from a 3-task smoke. Per-environment source counts at these pins include 11060 filtered I3 tasks, 454 filtered LiveCodeBench tasks, 1150 GraphWalks tasks, and MRCR bucket sizes 85 and 141. Full rollout settings exceed the upstream >500-rollout guideline.
To repeat the end-to-end check from a clean checkout:
nix develop . --no-write-lock-file
eval/scripts/setup
eval/scripts/data aime24
eval/scripts/sandbox
export QWEN_API_KEY_FILE=/absolute/path/to/serving/state/api-key
eval/scripts/run smoke
Opt-in task time and board energy #
The measurement path is eval/scripts/run PROFILE [TASKSET] --measure-power.
It wraps the same pinned native command without changing tasks, retries, rewards,
sampling, runtime or serving policy. Use an exclusive quiet endpoint and select
the actual serving GPU; no board limit or service is modified:
eval/scripts/run quick aime24 --measure-power --power-gpu=0 \
--power-interval-seconds=0.5 --power-max-seconds=86400 \
--power-max-samples=200000
eval/scripts/run --help documents the bounds. Collection stops at its duration
or sample bound; evaluation continues, with uncovered time explicitly reported.
Each nvidia-smi query has a two-second timeout. INT, TERM and HUP are forwarded
to the native process group; teardown has a ten-second grace period before kill,
and the collector is joined before evidence is written. SIGKILL cannot execute
cleanup. --dry-run cannot be combined with measurement. Native exit status and
raw rewards are retained; measurement failure is a warning and a separate status,
never an invented benchmark failure score or zero-joule observation.
The private run's TASKSET/measurement/ contains:
power-samples.json: selected board, host Unix/monotonic clock anchors, power in watts, each query's monotonic start/end and midpoint, and categorized read failures. These are host observations, not device timestamps.task-intervals.json: episode ordinals, operational status, original numeric reward components (including fractional scores), error counts, truncation and attributable trace-envelope intervals. No prompts or error text are copied.efficiency.json: sanitized per-environment machine-readable totals and denominators. Publish only this summary with separately reviewed deployment provenance; keep native traces and detailed records private.
Energy is the clipped trapezoidal integral between adjacent valid observations.
There is no extrapolation, idle subtraction, interpolation across failed reads,
or bridging gaps longer than three sample periods. observed_joules describes
only covered intervals; total_joules and joules/success are null unless the whole
native-command interval is covered. Coverage, gaps, read failures and collection
limits remain in the evidence. The scope is selected-board energy, not whole-host
or decode-only energy.
successful_task_rollouts counts single-trace episodes with both native
operational statuses true and the native weighted reward exactly 1; ok=true
alone is not solved. Fractional rewards remain fractional. Repeated rollouts are
not unique questions. Zero successes, unsupported/missing reward denominators,
or an interrupted/failed native CLI produce null efficiency ratios. Total wall
time and energy include startup, scoring, failed attempts, retry/backoff and
teardown, not just successful episodes.
At the pinned Verifiers revision, episodes have no serialized whole-episode start/end. Only each persisted trace's start and phase spans are available. Task attribution therefore uses persisted trace envelopes, never claims discarded retry or whole-episode durations, and reports unattributed overhead. Attribution requires resolved C1, nonoverlapping spans inside the measured run, and host Unix/monotonic clock drift no greater than 50 ms. Missing/invalid spans or clock jumps disable attribution explicitly without changing native scores or whole-run monotonic energy. This does not correct an unobserved clock jump between samples; query intervals and clock uncertainty are retained.
This path's implementation is not a new measurement result or qualification. Verification must exercise synthetic linear/clipped intervals, failed-read and long-gap coverage, zero/fractional/full rewards, missing or torn native traces, overlapping timestamps, clock jumps, collection bounds and signal cleanup before using a real isolated GPU run as efficiency evidence.