# Metrics and reporting

## Primary curriculum metric

**Frontier:** highest contiguous Project Euler problem that met the configured gate in every language.

Always report the gate beside the frontier, for example:

```text
frontier=17; languages=python,javascript,julia; fanout=16;
required=12 correct/language; consecutive_rounds=2;
regression_window=10; minimum_regression_rate=0.8
```

A frontier without its gate is not comparable.

## Evaluation metrics

Per problem, language, strategy, model, and checkpoint:

- attempts and completed attempts;
- exact correct count;
- empirical pass@1 (correct / attempts);
- observed pass@k (whether this finite fan-out had any success);
- boxed-format rate;
- successful write/edit rate;
- assigned-runtime direct-file execution attempt rate;
- zero-exit execution rate;
- mutated-file execution correlation rate;
- exact executed-output rate;
- semantic final-response answer rate;
- functional qualification rate (the conjunction above);
- latency median and distribution;
- input/output/cache tokens where the provider reports them;
- tool-call count and tool errors;
- escape attempts, actual containment breaches, and pre-audit score;
- timeout/error rate.

“Observed pass@k” is not an unbiased statistical estimator. Use a standard pass@k estimator only when samples are exchangeable and the assumptions are documented.

## Learning metrics

For Tinker runs:

- environment reward and exact correctness;
- format reward;
- reward variance within groups;
- policy/importance-sampling diagnostics;
- KL before/after updates when enabled;
- token counts and trajectory truncation;
- checkpoint path and step;
- current/replay problem mixture;
- language and strategy composition.

Read rollout summaries and log trees. A rising reward can still mean answer memorization, constant-print programs, malformed code, or exploitation of the parser.

## Sequential evaluation cautions

Repeatedly testing until a gate passes introduces optional stopping. The gate is an engineering mastery criterion, not a hypothesis-test p-value. Preserve every failed round rather than reporting only the passing round.

Changing model, adapter, prompt, language set, tools, thinking level, timeout, or scorer creates a new experimental condition. The evaluator resets consecutive passes when the model signature changes; other condition changes should use a new state file.

A scorer repair after sampling invalidates promotion from the original summary. Preserve the raw events and create a labeled retrospective re-score. Do not call it a fresh gate or use it to unlock the next problem.

## Transfer comparisons

Before comparing transfer runs, verify identical task digest, task selection, languages, fanout, model settings apart from checkpoint revision, sandbox image ID, evaluator/scorer/tool hashes, expected row count, and result/summary receipts. Report each language and task family beside the overall count. Small unpaired Fisher tests are descriptive; failure to reject a regression is not evidence that no regression exists.

## Generalization evidence

Strong claims require at least one of:

- first-attempt performance on untouched future Euler problems;
- hidden parameterized variants;
- held-out executable tests;
- transfer to a separate math/code benchmark.

Report curriculum mastery and generalization separately.