# Sequential Project Euler agents with Pi and Tinker This example is an evaluation-first research loop for improving a small model on computational mathematics. It combines two complementary systems: 1. **Pi harness evaluation** — fan out complete coding agents into isolated Docker workspaces, require each agent to solve a Project Euler problem in an assigned programming language, execute the exact file it changed, and score the observed program output and final response separately. 2. **Tinker GRPO** — sample groups from a trainable Tinker policy, score exact answers, update the policy, register the resulting checkpoint in Pi, and run the harness evaluation again. The curriculum is sequential. Problem `n + 1` stays locked until the current model repeatedly meets the configured threshold on problem `n` in every configured language and retains enough accuracy on recently mastered problems. ## Research questions - How far can a compact model progress through Project Euler? - Does GRPO improve exact-answer accuracy, not just training reward? - Do gains transfer to unseen future problems? - Which programming languages help or hurt a small model? - Which approach conditions produce useful exploration? - Does the model retain previously mastered problems? - How many rollouts, tokens, and training steps are needed per unlocked problem? ## Important distinctions ### Pi fan-outs are not themselves GRPO Pi can produce rich agent trajectories from arbitrary providers, prompts, and tools. Those are valuable observations and may be reviewed for SFT or analysis, subject to the provider's terms. GRPO, however, needs samples and policy log-probabilities from the trainable Tinker policy. Arbitrary external Pi traces cannot simply be treated as on-policy GRPO trajectories. The bridge is the shared problem manifest and scorer: ```text prepared problems + exact answers ├── Pi tool-using evaluation └── Tinker policy sampling / GRPO ``` ### Curriculum mastery is not the same as generalization A model repeatedly trained on a fixed Project Euler problem can memorize its answer. Executing code does not eliminate this failure mode: a program can print a constant. Therefore the reports distinguish: - **Curriculum mastery:** highest contiguous problem that passed the gate. - **Forward transfer:** performance on each problem before training on it. - **Regression:** performance on recently mastered problems. - **Generalization:** performance on untouched future problems or generated parameterized variants. Do not describe a higher curriculum frontier as general mathematical reasoning without transfer evidence. ## Dataset and license The local archive used while developing this example came from: - Dataset: [Project Euler Questions and Solutions](https://www.kaggle.com/datasets/dheerajmpai/projecteuler) - Creator: Dheeraj M Pai - Kaggle license: **CC BY-NC-SA 4.0** - Dataset description: educational and non-commercial use; it states that it was shared with permission from the Project Euler team. Machine is Apache-2.0, so this repository does **not** bundle or relicense the question/answer corpus. `prepare.py` imports a local archive into the ignored `data/` directory. Generated manifests include answers and must never be exposed to evaluated agents. The archive contains 849 problem HTML files and 840 parsed answers. Some problems reference external documents, and many include images. `prepare.py --fetch-resources` fetches referenced document files from Project Euler and caches them under ignored local data. Images are recorded but not automatically treated as required because some, such as problem 15's grid illustration, are redundant with the text. Audit image-dependent questions before making strong claims. See [docs/DATASET.md](docs/DATASET.md). ## Repository layout ```text examples/project-euler/ ├── README.md ├── config.example.toml ├── build_tool_bootstrap.py audited non-Euler tool-use corpus generator ├── prepare.py local Kaggle ZIP → answer-bearing manifest ├── evaluate.py concurrent Pi fan-outs and sequential gate ├── evaluate_transfer.py frozen unrelated multilingual transfer suite ├── compare_transfer.py receipt-checked transfer comparison ├── evaluate_tool_rl.py read-only hidden-variant checkpoint comparison ├── evaluate_tinker.py read-only native Tinker baseline sampling ├── replay_gate_v2.py retrospective functional-gate re-scoring ├── replay_transfer_v2.py retrospective transfer re-scoring ├── preflight_tool_rl.py no-spend hidden-variant and runtime preflight ├── score.py correlated mutation/execution/output signals ├── curriculum.py persistent mastery frontier ├── report.py JSON and Markdown aggregate reports ├── pi_sandbox_tools.ts Docker-confined Pi file and shell tools ├── train_grpo.py native Tinker GRPO scaffold ├── train_pi_grpo.py bounded Pi-shaped file/shell GRPO ├── train_tool_sft.py Qwen tool-protocol bootstrap SFT ├── train_tool_grpo.py sandbox-backed hidden-variant agent RL ├── tinker_proxy_provider.ts Pi provider for the renderer-aware local proxy ├── src/euler_common.py shared parsing and scoring primitives ├── src/euler_tinker.py shared native evaluation/training environment ├── src/euler_pi_rl.py Pi-shaped Tinker tool environment and reward ├── src/euler_variants.py deterministic variants and sandbox harnesses ├── src/euler_tool_rl.py Tinker agent-tool dataset and reward ├── src/pi_docker_sandbox.py pinned Docker lifecycle and receipts ├── src/pi_rl_tasks.py generated Pi-shaped train/eval tasks ├── src/transfer_tasks.py frozen unrelated transfer tasks ├── docs/ │ ├── DATASET.md │ ├── DESIGN.md │ ├── EXPERIMENTS.md │ └── METRICS.md └── tests/ ``` The following paths are ignored: ```text data/ questions, answers, resources, curriculum state runs/ raw Pi events, responses, metrics, checkpoints config.toml local provider/model configuration ``` ## Setup Requirements: - Python 3.11+ - Docker and the digest-pinned SandboxFusion image in `config.example.toml` - Pi installed and authenticated for the chosen inference provider - the provider/model visible in `pi --list-models` - Python, Node, and Julia for the default language matrix - Tinker credentials and Cookbook dependencies for `evaluate_tinker.py` and `train_grpo.py` From this directory: ```bash cp config.example.toml config.toml python3 prepare.py \ --archive ~/Downloads/euler-questions.zip \ --fetch-resources ``` The importer writes: ```text data/problems.jsonl id, prompt, answer, availability, resource references, source URL data/manifest.json provenance, license, counts, missing resources ``` **Never place `data/problems.jsonl` in an agent workspace.** The evaluator passes only the selected prompt and required resource files into a fresh temporary directory. ## First baseline For a cheap smoke test, keep `fanout_per_language = 2` in `config.toml`: ```bash python3 evaluate.py --config config.toml --mode canonical ``` The default matrix is: ```text problem 1 × {Python, JavaScript, Julia} × 2 rollouts ``` Each agent is instructed to: 1. use its assigned language; 2. write code to the assigned temporary workspace; 3. execute the code with Pi's bash tool; 4. explain the algorithm; 5. state the verified answer; boxed formatting remains a separate presentation metric. For a meaningful gate, use at least eight rollouts per language and require six correct: ```toml [evaluation] fanout_per_language = 8 [curriculum] required_correct_per_language = 6 required_consecutive_rounds = 2 ``` The same model must pass two consecutive rounds. Previously mastered problems are sampled as a regression window. Changing the provider/model/thinking signature resets consecutive passes. ## Sequential curriculum State lives in `data/curriculum-state.json` and begins at problem 1. A canonical round using `qualification_mode = "functional_v2"` passes when: - every configured language has the required number of functionally qualified trajectories; - each qualified trajectory successfully writes or edits a relative program file, executes that same file with the assigned runtime, exits zero, prints the expected value as its final nonempty output line, and states the same value in the final response; - inline `python -c`, `node -e`, Julia `-e`, and unrelated-file executions cannot borrow credit from a dummy write; - any audited escape attempt disqualifies the trajectory even when Docker blocked it; - the regression sample meets `minimum_regression_rate`; - all required rollouts completed. After the configured number of consecutive passing rounds: ```text highest_mastered = current_problem current_problem += 1 consecutive_passes = 0 ``` Unavailable problems do not get silently skipped. The evaluator stops and asks you to fetch or supply the missing resource. This preserves the meaning of “highest contiguous mastered problem.” Inspect state: ```bash python3 curriculum.py data/curriculum-state.json ``` ## Exploration fan-outs Canonical prompts are neutral and are the only prompts used for unlocking. Exploration prompts deliberately vary the approach: ```bash python3 evaluate.py --config config.toml --mode explore --fanout 2 ``` Configured strategy arms include: - derive the mathematics first; - brute force, then optimize; - tests first; - recurrence/invariant/dynamic programming; - readable and auditable code. Each `(problem, language, strategy)` is logged separately. For GRPO, responses with different prompts must not be centered as if they came from the same group. `train_grpo.py` creates one group per condition. ## Reports Every evaluator run writes: ```text runs// ├── manifest.json ├── results.jsonl ├── summary.json ├── raw/*.jsonl ├── raw/*.stderr.txt └── machine-data/ ``` The manifest is written before sampling, records the scorer/evaluator/tool hashes in the model signature, retains partial-result hashes on failure, and binds completed results and summary files by SHA-256. A scorer or prompt change therefore resets consecutive gate passes instead of inheriting state under the same model alias. Build an aggregate report: ```bash python3 report.py \ runs/*/results.jsonl \ --output-dir runs/aggregate ``` Read both `report.md` and raw responses. Numbers alone do not reveal answer memorization, tool failures, reward hacking, or invalid code. ## Docker isolation and functional gate v2 `evaluate.py` starts one digest-pinned container per rollout. The container has no network, a read-only root, all Linux capabilities dropped, `no-new-privileges`, explicit CPU/memory/PID limits, and only the temporary workspace mounted at `/workspace`. `pi_sandbox_tools.ts` routes `read`, `write`, `edit`, and `bash` through `docker exec`; Pi does not receive its session environment. Startup and timeout paths remove the preselected container name, and each row records the container ID, image ID, mount, and cleanup outcome. The lexical containment audit remains useful after isolation. It records direct absolute-path, home-path, and parent-traversal attempts. A blocked attempt is still an agent-behavior failure and disqualifies the rollout; `containment_breach` separately records a violation observed without the Docker boundary. Formatting does not determine functional correctness. Gate v2 records five independent axes: - file mutation; - execution of that mutated file; - zero exit status; - exact executed output; - semantic answer in the final response. Use `replay_gate_v2.py` only for diagnosis when an older trajectory set needs the corrected scorer. A scoring rule designed or repaired after observing trajectories cannot authorize promotion. Run a new frozen-condition gate after the scorer is final. ## Frozen transfer suite `evaluate_transfer.py` runs four non-Euler file tasks across Python, JavaScript, and Julia: filtered log aggregation, inventory valuation, repair of an off-by-one checksum program, and aggregation across file shards. The full task/resource/answer set has a fixed SHA-256 digest. Answers stay in the private run manifest and never enter the mounted workspace. ```bash python3 evaluate_transfer.py \ --config config.toml \ --fanout 2 ``` Completed manifests hash `results.jsonl` and `summary.json`, record expected and completed row counts, and retain source hashes for the evaluator, scorer, tools, and task generator. `compare_transfer.py` refuses to compare runs unless their task selection, languages, fanout, model settings apart from revision, sandbox image, source hashes, task records, row counts, and file receipts match. The suite is diagnostic unless a numerical acceptance rule was written before sampling. Freezing tasks is necessary; inventing the pass threshold after seeing the table is still moving the goalposts, just with better stationery. ## Native Tinker baseline Use the read-only evaluator to sample a base model or sampler checkpoint without creating training state or performing an optimizer update: ```bash ../../.venv/bin/python evaluate_tinker.py \ --problems 1 \ --model Qwen/Qwen3.5-4B \ --renderer qwen3_5_disable_thinking \ --max-tokens 4096 \ --fanout 8 ``` The evaluator uses the Tinker Cookbook benchmark runner, stores decoded trajectories under `runs/`, and reports exact correctness, required fenced-code format, truncation, pass@1/pass@k, and per-language conditions. It never executes generated code; Pi remains the complete agent/runtime benchmark. The first confirmed Qwen3.5-4B frontier was problem 1 under the command above: 17/24 exact, 23/24 formatted, and 0/24 truncated. JavaScript scored 7/8, Julia 5/8, and Python 5/8. This fails the serious 6/8-each-language gate and provides mixed GRPO groups without relying on truncation failures. Native success is not agent success. A two-step direct-answer checkpoint later passed two native 6/8-per-language rounds but failed the Pi harness 0/6 with zero tool calls. Never advance the full curriculum from native metrics alone; Pi execution remains the gate. Models without reliable tool behavior need bootstrap SFT or sandbox-backed agent RL. ## Tool-use bootstrap SFT Direct-answer training does not teach the Pi tool protocol. Build the deterministic non-Euler bootstrap corpus, inspect it, and run SFT before agent evaluation: ```bash python3 build_tool_bootstrap.py --output data/tool-bootstrap.jsonl ../../.venv/bin/python train_tool_sft.py \ data_file=data/tool-bootstrap.jsonl \ renderer_name=qwen3_5_disable_thinking \ behavior_if_log_dir_exists=delete ``` The generator creates 45 balanced Python, JavaScript, and Julia trajectories: 36 straightforward write/execute flows and nine write/fail/read/edit/re-execute recovery flows. Tasks cover squares, cubes, triangular numbers, Fibonacci, and weighted checksums. Every expected result is deterministic and locally executable. It contains no Project Euler prompts or answers. Qwen3.5 lacks the sequence-extension property required for safely training every assistant turn in one rendered sequence. The dataset therefore creates one full-prefix supervised example per assistant turn and uses `LAST_ASSISTANT_MESSAGE`. User messages and tool results remain context-only; tool calls, recovery explanations, and final assistant answers are targets. For a measured language-specific weakness, `language_multipliers=julia:3` oversamples only the training split; held-out rows are never duplicated. Treat this as protocol bootstrap, not task training. Register the resulting sampler and require real Pi `write`/`bash` calls before returning to Euler GRPO. Tinker's hosted Anthropic/OpenAI compatibility endpoints currently drop Qwen3.5 checkpoint XML tool calls. For full Pi evaluation, run the Cookbook's renderer-aware loopback proxy: ```bash ../../.venv/bin/python -m tinker_cookbook.capture.proxy.serve \ --host 127.0.0.1 \ --port 7462 \ --base-model Qwen/Qwen3.5-4B \ --model-path tinker://.../sampler_weights/final \ --renderer-name qwen3_5_disable_thinking \ --default-max-tokens 4096 ``` Configure the evaluator model as: ```toml [model] pi_command = "pi" pi_args = ["--no-extensions", "-e", "./tinker_proxy_provider.ts"] provider = "tinker-proxy" model = "checkpoint" thinking = "" revision = "checkpoint-alias-or-hash" ``` The proxy binds to loopback and uses the native SamplingClient plus Cookbook renderer to translate Pi tool schemas and parse checkpoint tool calls. `evaluate.py` accepts `model.pi_args` so the provider extension can be loaded explicitly. Non-loopback proxy binds require authentication and are not recommended for this example. ## Tinker GRPO Install the Machine Python environment as described in the repository root README. Then run a small smoke job: ```bash ../../.venv/bin/python train_grpo.py \ problem_id=1 \ model_name=Qwen/Qwen3.5-4B \ group_size=4 \ groups_per_batch=3 \ max_steps=2 \ behavior_if_log_dir_exists=delete ``` A more serious first run might use: ```bash ../../.venv/bin/python train_grpo.py \ problem_id=1 \ model_name=Qwen/Qwen3.5-4B \ renderer_name=qwen3_5_disable_thinking \ group_size=8 \ groups_per_batch=6 \ learning_rate=4e-5 \ max_tokens=4096 \ max_steps=20 ``` `train_grpo.py` scores exact boxed answers and language-specific code-block formatting. It does **not** execute generated code during RL. Start this answer-only path at a frontier with mixed stochastic rewards: all-good and all-bad groups provide little or no group-relative signal. Qwen3.5-4B also needs a substantial budget with its default thinking renderer. Controlled smoke runs still saturated 4,096 tokens; 8,192 was the first tested budget to produce some fully completed responses. Use the hidden-variant tool path below when execution belongs inside the learning environment. Never execute untrusted model-generated code directly on the host. ## Hidden-variant tool RL `train_tool_grpo.py` adds the first sandbox-backed agent RL path. It trains on two parameterized families rather than fixed public answers: - sum all values below `limit` divisible by either supplied divisor; - sum even Fibonacci values below a supplied limit. Each task contains a deterministic hidden case set. Train and evaluation tasks derive from separate seed namespaces, and the preflight rejects any overlapping arguments. The model receives the function contract and public examples but never receives hidden inputs or expected outputs in the conversation. `check_solution` runs the submitted Python, JavaScript, or Julia function through a loopback-only SandboxFusion endpoint and returns aggregate counts. The harness uses an unpredictable marker for each call, rejects duplicate result markers, and never forwards sampled stdout, stderr, compiler messages, or transport details to the policy. “Hidden” describes rollout visibility, not a permanent secret benchmark. The task generator and default seed are public and reproducible. Do not tune against the evaluation split or present it as external generalization evidence. The reward is `0.5 × hidden_pass_rate + 0.5 × all_cases_passed`. A missing tool call or infrastructure failure receives `-0.1`; a submitted program that fails to compile or run receives zero and a separate metric. This produces partial signal without rewarding a memorized Project Euler scalar. Each episode allows two tool calls so the model can repair one failed attempt from aggregate feedback. Run the no-spend preflight first: ```bash python3 preflight_tool_rl.py ``` The command executes bundled reference implementations through the local Python, Node.js, and Julia runtimes. It verifies deterministic train/eval separation, emits only task-set digests and aggregate receipts, and does not contact SandboxFusion or Tinker. To validate the same trusted references through SandboxFusion, start the Cookbook-compatible container separately, then run: ```bash ../../.venv/bin/python preflight_tool_rl.py --sandbox ``` The sandbox URL must resolve to loopback and end in `/run_code`. The code rejects remote endpoints because hidden cases should not leave the host. After both preflights pass, a bounded paid smoke run can start from the tool-SFT checkpoint: ```bash ../../.venv/bin/python train_tool_grpo.py \ load_checkpoint_path=tinker://.../sampler_weights/final \ max_steps=2 \ behavior_if_log_dir_exists=delete ``` The training command calls Tinker and incurs usage. A completed checkpoint remains a candidate: compare it with the incumbent on the frozen hidden evaluation split, the full Pi P1 gate, and unrelated tool-use tasks before promotion. Before any Tinker API request, the command runs one trusted program in each SandboxFusion language runtime. It writes `variant-manifest.json` with the Git revision, renderer, checkpoint, seed, task IDs, and train/evaluation digests before invoking training, then records whether the invocation failed or completed. Compare the source and candidate checkpoints without creating an optimizer: ```bash ../../.venv/bin/python evaluate_tool_rl.py \ --checkpoint source=tinker://.../sampler_weights/final \ --checkpoint candidate=tinker://.../sampler_weights/final \ --fanout 8 ``` The evaluator regenerates the same fixed internal split, requires its SHA-256 digest to match the frozen default, and gives each family/language condition eight independent trajectories per checkpoint. A deliberate task-configuration change must supply its new expected digest explicitly. It stores complete trajectory and token receipts under ignored `runs/`, reports full-case pass rates with Wilson intervals, and keeps partial hidden-case coverage as a diagnostic rather than counting it as pass@k success. It independently reconstructs binary pass@k from `hidden_all_passed` and fails if that differs from the benchmark runner. The evaluator creates sampling clients only; it does not create a training client or optimizer. ## Pi-shaped remediation GRPO `train_pi_grpo.py` is a bounded remediation path for failures that appear specifically in file/shell agent behavior. Its Tinker environment exposes `read`, `write`, `edit`, and `bash` schemas matching the Pi harness. Each trajectory receives a fresh in-memory workspace; the bash tool executes only one relative program file through loopback SandboxFusion. Reward separates mutation, execution, exact output, and final-response correctness. Submission errors remain model-visible in bounded form so the policy can repair code, matching normal Pi behavior. This differs intentionally from hidden-variant RL, where diagnostics and stdout are suppressed to protect hidden cases. The default run is Julia-heavy rather than Julia-only: ```bash ../../.venv/bin/python train_pi_grpo.py \ load_checkpoint_path=tinker://.../weights/final \ max_steps=2 \ behavior_if_log_dir_exists=raise ``` The language schedule contains three Julia groups for each Python and JavaScript group. Generated train and evaluation tasks use separate deterministic namespaces and frozen digests. The command requires an explicit source checkpoint, records its bounds before creating training state, and writes checkpoint paths plus per-language held-out behavior. A completed run is not a promotion. If held-out Julia remains zero or the full Pi gate regresses, reject the checkpoint rather than negotiating with the metric. After training, register the sampler checkpoint with pi-tinker and evaluate it through Pi: ```text /tinker use tinker://.../sampler_weights/final /machine student ``` Then update `config.toml` to the registered Tinker model and rerun `evaluate.py`. ## Recommended research loop ```text 1. Evaluate current frontier through Pi. 2. Save pre-training metrics as forward-transfer evidence. 3. If the gate passes, unlock the next problem. 4. If it fails, inspect responses and reward failures. 5. Run a small GRPO update on the current problem plus replay. 6. Register the checkpoint. 7. Re-evaluate current + regression problems through Pi. 8. Compare pass rates, execution rates, tokens, latency, KL, and regressions. 9. Repeat, preserving every config and checkpoint path. ``` `train_grpo.py` trains only the current fixed problem. Before long sequential runs, add replay of previously mastered problems to reduce catastrophic forgetting. ## Provider and data-policy warning A provider allowing coding use does not necessarily allow using its outputs to train a competing model. Review the provider's current terms before promoting Pi observations into SFT, preference, or distillation datasets. Keep automatic Machine captures as `observed` until a human makes both a quality judgment and a rights/provenance judgment. ## Safety and reproducibility - Pi evaluation workspaces are temporary and contain no answer key. - Pi file and shell operations run through a fresh Docker container with no network and only the temporary workspace mounted. - Host-side lexical and symlink checks select the mounted path; the actual read, write, edit, and command effects occur in the container. A symlink race can at worst resolve against the container filesystem, whose root is read-only, rather than exposing an arbitrary host path. - The lexical audit disqualifies direct escape attempts independently of whether the container blocked them. - Tinker code-RL uses a loopback SandboxFusion service; no sampled code runs on the host. - Keep model, provider, checkpoint, thinking level, temperature, concurrency, prompt strategy, and dataset manifest with every run. - Do not compare checkpoints using different gates or evaluator prompts without labeling the change. - Do not tune on a permanent held-out set. ## Current limitations - Project Euler is widely available online and likely present in model pretraining data. - Fixed answers allow memorization and constant-print reward hacking. - Image-dependent problems require manual audit. - Functional scoring accepts only a zero-exit assigned-runtime command that executes the same relative file previously written or edited. It does not prove the implementation is mathematically general rather than a constant-print program. - The lexical containment audit is bypassable as detection; Docker supplies the effect boundary. Do not mount answer-bearing or private host directories into the rollout container. - `train_grpo.py` remains answer-only. `train_tool_grpo.py` executes hidden variants through SandboxFusion but currently covers only two early Euler families. The fixed internal evaluation split is validation data, not an untouched external benchmark. - Repeated sequential testing creates optional-stopping effects; the mastery gate is an engineering criterion, not a p-value. - Strict progression can be blocked by a missing resource or genuinely difficult problem. That is intentional, but the state may be copied to start a separate ablation. See [docs/EXPERIMENTS.md](docs/EXPERIMENTS.md) for the staged plan, [docs/METRICS.md](docs/METRICS.md) for reporting rules, and [docs/INITIAL_SMOKE.md](docs/INITIAL_SMOKE.md) for the first real end-to-end result.