Design #

Goals #

• Make a Pi harness configuration empirically measurable.
• Support base Tinker models and registered adapter checkpoints.
• Fan out independent coding-agent attempts by language and approach.
• Use exact, deterministic rewards where possible.
• Advance through Project Euler in strict numerical order.
• Preserve raw trajectories and enough metadata to reproduce every score.
• Separate observed behavior from approved training data.

Non-goals #

• Claim that fixed Project Euler accuracy proves broad mathematical reasoning.
• Treat arbitrary provider output as automatically licensed training data.
• Treat Pi event traces as on-policy GRPO data.
• Execute untrusted generated code on a production host.

Components #

Importer #

prepare.py reads a local Kaggle ZIP without extracting it into the repository. It parses question HTML, the answer list, resource references, source URLs, and availability. Answer-bearing output stays in ignored local storage.

Pi evaluator #

evaluate.py creates a fresh temporary directory and digest-pinned Docker container per rollout. It copies only required problem resources, never the answer manifest. The container has no network, a read-only root, dropped capabilities, resource limits, and only that workspace mounted at /workspace. Pi runs in JSON mode with an ephemeral session and a restricted tool list. Its read, write, edit, and bash operations execute through the container.

The evaluator also audits explicit absolute-path, home-path, and parent-traversal tool calls. The audit is behavior detection rather than the effect boundary. Any attempt disqualifies a trajectory even when Docker prevented it. Each row retains the pre-audit score, attempted path, container/image receipt, and cleanup outcome.

Pi is the system-under-test here: model, system prompt, context, tools, and model adapter all affect results.

Scorer #

Functional gate v2 separates presentation from behavior. A qualified trajectory must mutate a relative program file, execute that same file with the assigned runtime, exit zero, print the normalized expected value as its final nonempty output line, and state that value in its final response. The scorer rejects inline runtime flags and unrelated-file executions. Boxed and FINAL_ANSWER: formats remain presentation diagnostics.

The exact output requirement blocks a command whose error text merely contains the answer. Correlating the executed path with the mutation blocks a dummy write followed by python -c 'print(answer)'. This still does not prove generality: a mutated file may contain a constant.

Curriculum #

The state machine stores the current problem, highest mastered contiguous problem, consecutive passing rounds, model signature, and gate history. Model changes reset consecutive passes. Missing questions/resources stop progression rather than being skipped.

Reports #

Raw rollout rows are the source of truth. Markdown summaries report empirical pass fractions, observed pass@k, execution rate, and median latency by problem/language/phase.

Frozen transfer evaluation #

evaluate_transfer.py runs four non-Euler file tasks across the same language matrix. The task/resource/answer set has a fixed digest. Completed manifests bind the selection, fanout, sandbox image, evaluator/scorer/tool hashes, expected row count, result hash, and summary hash. compare_transfer.py rejects mismatched manifests before computing descriptive counts and exact Fisher tests. Transfer does not authorize promotion unless its numerical acceptance rule was fixed before sampling.

Tinker GRPO #

train_grpo.py samples the trainable policy directly. Each problem/language/strategy condition is an independent group. Reward is exact answer correctness with a small code-format penalty. The initial implementation is answer-only with code in the response.

Tool-protocol bootstrap #

Native answer generation and Pi agent behavior are separate capabilities. A model can score well in the native environment while never emitting a structured tool call through Pi. The synthetic bootstrap corpus teaches write, read, edit, bash, tool-result interpretation, and recovery without leaking Euler answers.

For Qwen3.5, render one full conversation prefix per assistant target with LAST_ASSISTANT_MESSAGE. Do not use ALL_ASSISTANT_MESSAGES: its renderer lacks the sequence-extension property, so earlier assistant turns would receive an incorrect token prefix.

Why group by language and strategy #

GRPO centers advantages within a group. Comparing a Python prompt against a Julia prompt as though they were identical confounds prompt difficulty with response quality. Therefore each group contains multiple stochastic responses to one exact condition.

Sandbox-backed hidden-variant RL #

train_tool_grpo.py follows Tinker Cookbook's recipes/code_rl architecture. Each (family, hidden case set, language) condition becomes one group. The renderer creates a tool-aware conversation prefix, build_agent_tool_env runs the multi-turn policy, and check_solution sends submitted code to a loopback SandboxFusion service.

The initial task set parameterizes the mathematical families behind Euler problems 1 and 2. Deterministic seed derivation creates separate train and evaluation namespaces. The prompt includes a function contract and public examples; the sandbox harness appends hidden calls after the submitted function and returns only aggregate pass counts. It assigns an unpredictable result marker per call, rejects duplicate markers, and suppresses all submitted stdout, stderr, compiler text, and transport details. Fixed public answers never enter the model-visible conversation.

Reward combines partial hidden-case coverage with an all-cases bonus. The tool permits one repair attempt and records submission failures separately from execution-infrastructure failures. This gives GRPO group-relative signal without returning hidden-case diagnostics to the policy.

The first implementation supports Python, JavaScript, and Julia through SandboxFusion. It rejects non-loopback sandbox URLs. The no-spend preflight executes only bundled trusted reference code on the host; sampled model code must never run there.

Future task families should add stronger hidden tests and a second sandbox backend. A task family is not admitted merely because its fixed Euler answer can be checked.

Pi-shaped file/shell RL #

train_pi_grpo.py addresses a narrower failure: a checkpoint may understand a task but mishandle Pi's file and shell loop. The environment exposes the same four tool names, keeps one fresh workspace per trajectory, and permits only an assigned runtime plus one relative program path. SandboxFusion receives the current workspace files and returns bounded stdout/stderr for code repair.

The generated task families cover filtered logs, signed inventory arithmetic, weighted checksums, and file-shard aggregation. Train and evaluation tasks use disjoint deterministic namespaces. The default language schedule weights Julia three times relative to Python and JavaScript, while held-out evaluation remains balanced. Reward records mutation, successful execution, exact output, response answer, and their conjunction. A hard step/tool/token bound is part of the run manifest.

Unlike hidden-variant RL, this environment intentionally returns ordinary program output and bounded compiler errors because output interpretation and repair are the target behavior. Its held-out set is still internal validation. Full Docker-backed Pi evaluation remains the promotion gate.

Replay and forgetting #

Strict one-problem-at-a-time training is vulnerable to catastrophic forgetting. A mature curriculum batch should include:

• most groups from the current frontier;
• replay groups sampled from mastered problems;
• optional strategy/language balancing;
• an evaluation-only regression window.

A starting mix is 70% current problem, 20% recent mastered problems, and 10% older mastered problems. Report this mixture with every run.

Generalization extensions #

Fixed Euler tasks are contaminated and memorization-prone. Add:

• parameterized isomorphic variants with hidden inputs;
• held-out variants never used for gradients;
• code tests rather than one fixed scalar answer;
• transfer evaluation on later untouched Euler problems;
• external math/code benchmarks.

The sequential frontier remains useful as a curriculum/game metric, but generalization claims should use these extensions.