Pi harness evaluation — fan out complete coding agents into isolated Docker workspaces, require each agent to solve a Project Euler problem in an assigned programming language, execute the exact file it changed, and score the observed program output and final response separately.
Tinker GRPO — sample groups from a trainable Tinker policy, score exact answers, update the policy, register the resulting checkpoint in Pi, and run the harness evaluation again.

The curriculum is sequential. Problem n + 1 stays locked until the current model repeatedly meets the configured threshold on problem n in every configured language and retains enough accuracy on recently mastered problems.

Research questions #

How far can a compact model progress through Project Euler?
Does GRPO improve exact-answer accuracy, not just training reward?
Do gains transfer to unseen future problems?
Which programming languages help or hurt a small model?
Which approach conditions produce useful exploration?
Does the model retain previously mastered problems?
How many rollouts, tokens, and training steps are needed per unlocked problem?

Important distinctions #

Pi fan-outs are not themselves GRPO #

Pi can produce rich agent trajectories from arbitrary providers, prompts, and tools. Those are valuable observations and may be reviewed for SFT or analysis, subject to the provider's terms. GRPO, however, needs samples and policy log-probabilities from the trainable Tinker policy. Arbitrary external Pi traces cannot simply be treated as on-policy GRPO trajectories.

The bridge is the shared problem manifest and scorer:

prepared problems + exact answers
          ├── Pi tool-using evaluation
          └── Tinker policy sampling / GRPO

Curriculum mastery is not the same as generalization #

A model repeatedly trained on a fixed Project Euler problem can memorize its answer. Executing code does not eliminate this failure mode: a program can print a constant. Therefore the reports distinguish:

Curriculum mastery: highest contiguous problem that passed the gate.
Forward transfer: performance on each problem before training on it.
Regression: performance on recently mastered problems.
Generalization: performance on untouched future problems or generated parameterized variants.

Do not describe a higher curriculum frontier as general mathematical reasoning without transfer evidence.

Dataset and license #

The local archive used while developing this example came from:

Dataset: Project Euler Questions and Solutions
Creator: Dheeraj M Pai
Kaggle license: CC BY-NC-SA 4.0
Dataset description: educational and non-commercial use; it states that it was shared with permission from the Project Euler team.

Machine is Apache-2.0, so this repository does not bundle or relicense the question/answer corpus. prepare.py imports a local archive into the ignored data/ directory. Generated manifests include answers and must never be exposed to evaluated agents.

The archive contains 849 problem HTML files and 840 parsed answers. Some problems reference external documents, and many include images. prepare.py --fetch-resources fetches referenced document files from Project Euler and caches them under ignored local data. Images are recorded but not automatically treated as required because some, such as problem 15's grid illustration, are redundant with the text. Audit image-dependent questions before making strong claims.

See docs/DATASET.md.

Repository layout #

examples/project-euler/
├── README.md
├── config.example.toml
├── build_tool_bootstrap.py    audited non-Euler tool-use corpus generator
├── prepare.py                 local Kaggle ZIP → answer-bearing manifest
├── evaluate.py                concurrent Pi fan-outs and sequential gate
├── evaluate_transfer.py       frozen unrelated multilingual transfer suite
├── compare_transfer.py        receipt-checked transfer comparison
├── evaluate_tool_rl.py        read-only hidden-variant checkpoint comparison
├── evaluate_tinker.py         read-only native Tinker baseline sampling
├── replay_gate_v2.py          retrospective functional-gate re-scoring
├── replay_transfer_v2.py      retrospective transfer re-scoring
├── preflight_tool_rl.py       no-spend hidden-variant and runtime preflight
├── score.py                   correlated mutation/execution/output signals
├── curriculum.py              persistent mastery frontier
├── report.py                  JSON and Markdown aggregate reports
├── pi_sandbox_tools.ts        Docker-confined Pi file and shell tools
├── train_grpo.py              native Tinker GRPO scaffold
├── train_pi_grpo.py           bounded Pi-shaped file/shell GRPO
├── train_tool_sft.py          Qwen tool-protocol bootstrap SFT
├── train_tool_grpo.py         sandbox-backed hidden-variant agent RL
├── tinker_proxy_provider.ts   Pi provider for the renderer-aware local proxy
├── src/euler_common.py        shared parsing and scoring primitives
├── src/euler_tinker.py        shared native evaluation/training environment
├── src/euler_pi_rl.py         Pi-shaped Tinker tool environment and reward
├── src/euler_variants.py      deterministic variants and sandbox harnesses
├── src/euler_tool_rl.py       Tinker agent-tool dataset and reward
├── src/pi_docker_sandbox.py   pinned Docker lifecycle and receipts
├── src/pi_rl_tasks.py         generated Pi-shaped train/eval tasks
├── src/transfer_tasks.py      frozen unrelated transfer tasks
├── docs/
│   ├── DATASET.md
│   ├── DESIGN.md
│   ├── EXPERIMENTS.md
│   └── METRICS.md
└── tests/

The following paths are ignored:

data/       questions, answers, resources, curriculum state
runs/       raw Pi events, responses, metrics, checkpoints
config.toml local provider/model configuration

Setup #

Requirements:

Python 3.11+
Docker and the digest-pinned SandboxFusion image in config.example.toml
Pi installed and authenticated for the chosen inference provider
the provider/model visible in pi --list-models
Python, Node, and Julia for the default language matrix
Tinker credentials and Cookbook dependencies for evaluate_tinker.py and train_grpo.py

From this directory:

cp config.example.toml config.toml
python3 prepare.py \
  --archive ~/Downloads/euler-questions.zip \
  --fetch-resources

The importer writes:

data/problems.jsonl
  id, prompt, answer, availability, resource references, source URL

data/manifest.json
  provenance, license, counts, missing resources

Never place data/problems.jsonl in an agent workspace. The evaluator passes only the selected prompt and required resource files into a fresh temporary directory.

First baseline #

For a cheap smoke test, keep fanout_per_language = 2 in config.toml:

python3 evaluate.py --config config.toml --mode canonical

The default matrix is:

problem 1 × {Python, JavaScript, Julia} × 2 rollouts

Each agent is instructed to:

use its assigned language;
write code to the assigned temporary workspace;
execute the code with Pi's bash tool;
explain the algorithm;
state the verified answer; boxed formatting remains a separate presentation metric.

For a meaningful gate, use at least eight rollouts per language and require six correct:

[evaluation]
fanout_per_language = 8

[curriculum]
required_correct_per_language = 6
required_consecutive_rounds = 2

The same model must pass two consecutive rounds. Previously mastered problems are sampled as a regression window. Changing the provider/model/thinking signature resets consecutive passes.

Sequential curriculum #

State lives in data/curriculum-state.json and begins at problem 1.

A canonical round using qualification_mode = "functional_v2" passes when:

every configured language has the required number of functionally qualified trajectories;
each qualified trajectory successfully writes or edits a relative program file, executes that same file with the assigned runtime, exits zero, prints the expected value as its final nonempty output line, and states the same value in the final response;
inline python -c, node -e, Julia -e, and unrelated-file executions cannot borrow credit from a dummy write;
any audited escape attempt disqualifies the trajectory even when Docker blocked it;
the regression sample meets minimum_regression_rate;
all required rollouts completed.

After the configured number of consecutive passing rounds:

highest_mastered = current_problem
current_problem += 1
consecutive_passes = 0

Unavailable problems do not get silently skipped. The evaluator stops and asks you to fetch or supply the missing resource. This preserves the meaning of “highest contiguous mastered problem.”

Inspect state:

python3 curriculum.py data/curriculum-state.json

Exploration fan-outs #

Canonical prompts are neutral and are the only prompts used for unlocking. Exploration prompts deliberately vary the approach:

python3 evaluate.py --config config.toml --mode explore --fanout 2

Configured strategy arms include:

derive the mathematics first;
brute force, then optimize;
tests first;
recurrence/invariant/dynamic programming;
readable and auditable code.

Each (problem, language, strategy) is logged separately. For GRPO, responses with different prompts must not be centered as if they came from the same group. train_grpo.py creates one group per condition.

Reports #

Every evaluator run writes:

runs/<run>/
├── manifest.json
├── results.jsonl
├── summary.json
├── raw/*.jsonl
├── raw/*.stderr.txt
└── machine-data/

The manifest is written before sampling, records the scorer/evaluator/tool hashes in the model signature, retains partial-result hashes on failure, and binds completed results and summary files by SHA-256. A scorer or prompt change therefore resets consecutive gate passes instead of inheriting state under the same model alias.

Build an aggregate report:

python3 report.py \
  runs/*/results.jsonl \
  --output-dir runs/aggregate

Read both report.md and raw responses. Numbers alone do not reveal answer memorization, tool failures, reward hacking, or invalid code.

Docker isolation and functional gate v2 #

evaluate.py starts one digest-pinned container per rollout. The container has no network, a read-only root, all Linux capabilities dropped, no-new-privileges, explicit CPU/memory/PID limits, and only the temporary workspace mounted at /workspace. pi_sandbox_tools.ts routes read, write, edit, and bash through docker exec; Pi does not receive its session environment. Startup and timeout paths remove the preselected container name, and each row records the container ID, image ID, mount, and cleanup outcome.

The lexical containment audit remains useful after isolation. It records direct absolute-path, home-path, and parent-traversal attempts. A blocked attempt is still an agent-behavior failure and disqualifies the rollout; containment_breach separately records a violation observed without the Docker boundary.

Formatting does not determine functional correctness. Gate v2 records five independent axes:

file mutation;
execution of that mutated file;
zero exit status;
exact executed output;
semantic answer in the final response.

Use replay_gate_v2.py only for diagnosis when an older trajectory set needs the corrected scorer. A scoring rule designed or repaired after observing trajectories cannot authorize promotion. Run a new frozen-condition gate after the scorer is final.

Frozen transfer suite #

evaluate_transfer.py runs four non-Euler file tasks across Python, JavaScript, and Julia: filtered log aggregation, inventory valuation, repair of an off-by-one checksum program, and aggregation across file shards. The full task/resource/answer set has a fixed SHA-256 digest. Answers stay in the private run manifest and never enter the mounted workspace.

python3 evaluate_transfer.py \
  --config config.toml \
  --fanout 2

Completed manifests hash results.jsonl and summary.json, record expected and completed row counts, and retain source hashes for the evaluator, scorer, tools, and task generator. compare_transfer.py refuses to compare runs unless their task selection, languages, fanout, model settings apart from revision, sandbox image, source hashes, task records, row counts, and file receipts match.

The suite is diagnostic unless a numerical acceptance rule was written before sampling. Freezing tasks is necessary; inventing the pass threshold after seeing the table is still moving the goalposts, just with better stationery.

Native Tinker baseline #

Use the read-only evaluator to sample a base model or sampler checkpoint without creating training state or performing an optimizer update:

../../.venv/bin/python evaluate_tinker.py \
  --problems 1 \
  --model Qwen/Qwen3.5-4B \
  --renderer qwen3_5_disable_thinking \
  --max-tokens 4096 \
  --fanout 8

The evaluator uses the Tinker Cookbook benchmark runner, stores decoded trajectories under runs/, and reports exact correctness, required fenced-code format, truncation, pass@1/pass@k, and per-language conditions. It never executes generated code; Pi remains the complete agent/runtime benchmark.

The first confirmed Qwen3.5-4B frontier was problem 1 under the command above: 17/24 exact, 23/24 formatted, and 0/24 truncated. JavaScript scored 7/8, Julia 5/8, and Python 5/8. This fails the serious 6/8-each-language gate and provides mixed GRPO groups without relying on truncation failures.

Native success is not agent success. A two-step direct-answer checkpoint later passed two native 6/8-per-language rounds but failed the Pi harness 0/6 with zero tool calls. Never advance the full curriculum from native metrics alone; Pi execution remains the gate. Models without reliable tool behavior need bootstrap SFT or sandbox-backed agent RL.

Tool-use bootstrap SFT #

Direct-answer training does not teach the Pi tool protocol. Build the deterministic non-Euler bootstrap corpus, inspect it, and run SFT before agent evaluation:

python3 build_tool_bootstrap.py --output data/tool-bootstrap.jsonl
../../.venv/bin/python train_tool_sft.py \
  data_file=data/tool-bootstrap.jsonl \
  renderer_name=qwen3_5_disable_thinking \
  behavior_if_log_dir_exists=delete

The generator creates 45 balanced Python, JavaScript, and Julia trajectories: 36 straightforward write/execute flows and nine write/fail/read/edit/re-execute recovery flows. Tasks cover squares, cubes, triangular numbers, Fibonacci, and weighted checksums. Every expected result is deterministic and locally executable. It contains no Project Euler prompts or answers.

Qwen3.5 lacks the sequence-extension property required for safely training every assistant turn in one rendered sequence. The dataset therefore creates one full-prefix supervised example per assistant turn and uses LAST_ASSISTANT_MESSAGE. User messages and tool results remain context-only; tool calls, recovery explanations, and final assistant answers are targets. For a measured language-specific weakness, language_multipliers=julia:3 oversamples only the training split; held-out rows are never duplicated.

Treat this as protocol bootstrap, not task training. Register the resulting sampler and require real Pi write/bash calls before returning to Euler GRPO.

Tinker's hosted Anthropic/OpenAI compatibility endpoints currently drop Qwen3.5 checkpoint XML tool calls. For full Pi evaluation, run the Cookbook's renderer-aware loopback proxy:

../../.venv/bin/python -m tinker_cookbook.capture.proxy.serve \
  --host 127.0.0.1 \
  --port 7462 \
  --base-model Qwen/Qwen3.5-4B \
  --model-path tinker://.../sampler_weights/final \
  --renderer-name qwen3_5_disable_thinking \
  --default-max-tokens 4096

Configure the evaluator model as:

[model]
pi_command = "pi"
pi_args = ["--no-extensions", "-e", "./tinker_proxy_provider.ts"]
provider = "tinker-proxy"
model = "checkpoint"
thinking = ""
revision = "checkpoint-alias-or-hash"

The proxy binds to loopback and uses the native SamplingClient plus Cookbook renderer to translate Pi tool schemas and parse checkpoint tool calls. evaluate.py accepts model.pi_args so the provider extension can be loaded explicitly. Non-loopback proxy binds require authentication and are not recommended for this example.

Tinker GRPO #

Install the Machine Python environment as described in the repository root README. Then run a small smoke job:

../../.venv/bin/python train_grpo.py \
  problem_id=1 \
  model_name=Qwen/Qwen3.5-4B \
  group_size=4 \
  groups_per_batch=3 \
  max_steps=2 \
  behavior_if_log_dir_exists=delete

A more serious first run might use:

../../.venv/bin/python train_grpo.py \
  problem_id=1 \
  model_name=Qwen/Qwen3.5-4B \
  renderer_name=qwen3_5_disable_thinking \
  group_size=8 \
  groups_per_batch=6 \
  learning_rate=4e-5 \
  max_tokens=4096 \
  max_steps=20

train_grpo.py scores exact boxed answers and language-specific code-block formatting. It does not execute generated code during RL. Start this answer-only path at a frontier with mixed stochastic rewards: all-good and all-bad groups provide little or no group-relative signal. Qwen3.5-4B also needs a substantial budget with its default thinking renderer. Controlled smoke runs still saturated 4,096 tokens; 8,192 was the first tested budget to produce some fully completed responses. Use the hidden-variant tool path below when execution belongs inside the learning environment. Never execute untrusted model-generated code directly on the host.

Hidden-variant tool RL #

train_tool_grpo.py adds the first sandbox-backed agent RL path. It trains on two parameterized families rather than fixed public answers:

sum all values below limit divisible by either supplied divisor;
sum even Fibonacci values below a supplied limit.

Each task contains a deterministic hidden case set. Train and evaluation tasks derive from separate seed namespaces, and the preflight rejects any overlapping arguments. The model receives the function contract and public examples but never receives hidden inputs or expected outputs in the conversation. check_solution runs the submitted Python, JavaScript, or Julia function through a loopback-only SandboxFusion endpoint and returns aggregate counts. The harness uses an unpredictable marker for each call, rejects duplicate result markers, and never forwards sampled stdout, stderr, compiler messages, or transport details to the policy.

“Hidden” describes rollout visibility, not a permanent secret benchmark. The task generator and default seed are public and reproducible. Do not tune against the evaluation split or present it as external generalization evidence.

The reward is 0.5 × hidden_pass_rate + 0.5 × all_cases_passed. A missing tool call or infrastructure failure receives -0.1; a submitted program that fails to compile or run receives zero and a separate metric. This produces partial signal without rewarding a memorized Project Euler scalar. Each episode allows two tool calls so the model can repair one failed attempt from aggregate feedback.

Run the no-spend preflight first:

python3 preflight_tool_rl.py

The command executes bundled reference implementations through the local Python, Node.js, and Julia runtimes. It verifies deterministic train/eval separation, emits only task-set digests and aggregate receipts, and does not contact SandboxFusion or Tinker.

To validate the same trusted references through SandboxFusion, start the Cookbook-compatible container separately, then run:

../../.venv/bin/python preflight_tool_rl.py --sandbox

The sandbox URL must resolve to loopback and end in /run_code. The code rejects remote endpoints because hidden cases should not leave the host.

After both preflights pass, a bounded paid smoke run can start from the tool-SFT checkpoint:

../../.venv/bin/python train_tool_grpo.py \
  load_checkpoint_path=tinker://.../sampler_weights/final \
  max_steps=2 \
  behavior_if_log_dir_exists=delete

The training command calls Tinker and incurs usage. A completed checkpoint remains a candidate: compare it with the incumbent on the frozen hidden evaluation split, the full Pi P1 gate, and unrelated tool-use tasks before promotion.

Before any Tinker API request, the command runs one trusted program in each SandboxFusion language runtime. It writes variant-manifest.json with the Git revision, renderer, checkpoint, seed, task IDs, and train/evaluation digests before invoking training, then records whether the invocation failed or completed.

Compare the source and candidate checkpoints without creating an optimizer:

../../.venv/bin/python evaluate_tool_rl.py \
  --checkpoint source=tinker://.../sampler_weights/final \
  --checkpoint candidate=tinker://.../sampler_weights/final \
  --fanout 8

The evaluator regenerates the same fixed internal split, requires its SHA-256 digest to match the frozen default, and gives each family/language condition eight independent trajectories per checkpoint. A deliberate task-configuration change must supply its new expected digest explicitly. It stores complete trajectory and token receipts under ignored runs/, reports full-case pass rates with Wilson intervals, and keeps partial hidden-case coverage as a diagnostic rather than counting it as pass@k success. It independently reconstructs binary pass@k from hidden_all_passed and fails if that differs from the benchmark runner. The evaluator creates sampling clients only; it does not create a training client or optimizer.

Pi-shaped remediation GRPO #

train_pi_grpo.py is a bounded remediation path for failures that appear specifically in file/shell agent behavior. Its Tinker environment exposes read, write, edit, and bash schemas matching the Pi harness. Each trajectory receives a fresh in-memory workspace; the bash tool executes only one relative program file through loopback SandboxFusion. Reward separates mutation, execution, exact output, and final-response correctness. Submission errors remain model-visible in bounded form so the policy can repair code, matching normal Pi behavior. This differs intentionally from hidden-variant RL, where diagnostics and stdout are suppressed to protect hidden cases.

The default run is Julia-heavy rather than Julia-only:

../../.venv/bin/python train_pi_grpo.py \
  load_checkpoint_path=tinker://.../weights/final \
  max_steps=2 \
  behavior_if_log_dir_exists=raise

The language schedule contains three Julia groups for each Python and JavaScript group. Generated train and evaluation tasks use separate deterministic namespaces and frozen digests. The command requires an explicit source checkpoint, records its bounds before creating training state, and writes checkpoint paths plus per-language held-out behavior. A completed run is not a promotion. If held-out Julia remains zero or the full Pi gate regresses, reject the checkpoint rather than negotiating with the metric.

After training, register the sampler checkpoint with pi-tinker and evaluate it through Pi:

/tinker use tinker://.../sampler_weights/final
/machine student

Then update config.toml to the registered Tinker model and rerun evaluate.py.

Recommended research loop #

1. Evaluate current frontier through Pi.
2. Save pre-training metrics as forward-transfer evidence.
3. If the gate passes, unlock the next problem.
4. If it fails, inspect responses and reward failures.
5. Run a small GRPO update on the current problem plus replay.
6. Register the checkpoint.
7. Re-evaluate current + regression problems through Pi.
8. Compare pass rates, execution rates, tokens, latency, KL, and regressions.
9. Repeat, preserving every config and checkpoint path.

train_grpo.py trains only the current fixed problem. Before long sequential runs, add replay of previously mastered problems to reduce catastrophic forgetting.

Provider and data-policy warning #

A provider allowing coding use does not necessarily allow using its outputs to train a competing model. Review the provider's current terms before promoting Pi observations into SFT, preference, or distillation datasets. Keep automatic Machine captures as observed until a human makes both a quality judgment and a rights/provenance judgment.

Safety and reproducibility #

Pi evaluation workspaces are temporary and contain no answer key.
Pi file and shell operations run through a fresh Docker container with no network and only the temporary workspace mounted.
Host-side lexical and symlink checks select the mounted path; the actual read, write, edit, and command effects occur in the container. A symlink race can at worst resolve against the container filesystem, whose root is read-only, rather than exposing an arbitrary host path.
The lexical audit disqualifies direct escape attempts independently of whether the container blocked them.
Tinker code-RL uses a loopback SandboxFusion service; no sampled code runs on the host.
Keep model, provider, checkpoint, thinking level, temperature, concurrency, prompt strategy, and dataset manifest with every run.
Do not compare checkpoints using different gates or evaluator prompts without labeling the change.
Do not tune on a permanent held-out set.

Current limitations #

Project Euler is widely available online and likely present in model pretraining data.
Fixed answers allow memorization and constant-print reward hacking.
Image-dependent problems require manual audit.
Functional scoring accepts only a zero-exit assigned-runtime command that executes the same relative file previously written or edited. It does not prove the implementation is mathematically general rather than a constant-print program.
The lexical containment audit is bypassable as detection; Docker supplies the effect boundary. Do not mount answer-bearing or private host directories into the rollout container.
train_grpo.py remains answer-only. train_tool_grpo.py executes hidden variants through SandboxFusion but currently covers only two early Euler families. The fixed internal evaluation split is validation data, not an untouched external benchmark.
Repeated sequential testing creates optional-stopping effects; the mastery gate is an engineering criterion, not a p-value.
Strict progression can be blocked by a missing resource or genuinely difficult problem. That is intentional, but the state may be copied to start a separate ablation.

See docs/EXPERIMENTS.md for the staged plan, docs/METRICS.md for reporting rules, and docs/INITIAL_SMOKE.md for the first real end-to-end result.