machine
Parametric learning in your code harness.
Star 1
Fork 0
Watch
machine examples project-euler docs METRICS.md
3.4 kB · 78 lines
Markdown
at letta-code

Metrics and reporting #

Primary curriculum metric #

Frontier: highest contiguous Project Euler problem that met the configured gate in every language.

Always report the gate beside the frontier, for example:

frontier=17; languages=python,javascript,julia; fanout=16;
required=12 correct/language; consecutive_rounds=2;
regression_window=10; minimum_regression_rate=0.8

A frontier without its gate is not comparable.

Evaluation metrics #

Per problem, language, strategy, model, and checkpoint:

  • attempts and completed attempts;
  • exact correct count;
  • empirical pass@1 (correct / attempts);
  • observed pass@k (whether this finite fan-out had any success);
  • boxed-format rate;
  • successful write/edit rate;
  • assigned-runtime direct-file execution attempt rate;
  • zero-exit execution rate;
  • mutated-file execution correlation rate;
  • exact executed-output rate;
  • semantic final-response answer rate;
  • functional qualification rate (the conjunction above);
  • latency median and distribution;
  • input/output/cache tokens where the provider reports them;
  • tool-call count and tool errors;
  • escape attempts, actual containment breaches, and pre-audit score;
  • timeout/error rate.

“Observed pass@k” is not an unbiased statistical estimator. Use a standard pass@k estimator only when samples are exchangeable and the assumptions are documented.

Learning metrics #

For Tinker runs:

  • environment reward and exact correctness;
  • format reward;
  • reward variance within groups;
  • policy/importance-sampling diagnostics;
  • KL before/after updates when enabled;
  • token counts and trajectory truncation;
  • checkpoint path and step;
  • current/replay problem mixture;
  • language and strategy composition.

Read rollout summaries and log trees. A rising reward can still mean answer memorization, constant-print programs, malformed code, or exploitation of the parser.

Sequential evaluation cautions #

Repeatedly testing until a gate passes introduces optional stopping. The gate is an engineering mastery criterion, not a hypothesis-test p-value. Preserve every failed round rather than reporting only the passing round.

Changing model, adapter, prompt, language set, tools, thinking level, timeout, or scorer creates a new experimental condition. The evaluator resets consecutive passes when the model signature changes; other condition changes should use a new state file.

A scorer repair after sampling invalidates promotion from the original summary. Preserve the raw events and create a labeled retrospective re-score. Do not call it a fresh gate or use it to unlock the next problem.

Transfer comparisons #

Before comparing transfer runs, verify identical task digest, task selection, languages, fanout, model settings apart from checkpoint revision, sandbox image ID, evaluator/scorer/tool hashes, expected row count, and result/summary receipts. Report each language and task family beside the overall count. Small unpaired Fisher tests are descriptive; failure to reject a regression is not evidence that no regression exists.

Generalization evidence #

Strong claims require at least one of:

  • first-attempt performance on untouched future Euler problems;
  • hidden parameterized variants;
  • held-out executable tests;
  • transfer to a separate math/code benchmark.

Report curriculum mastery and generalization separately.