odd eval tasks — Jane Street library edition #

One directory per task; the model's input is the prompt file, run with the task directory (or a copy) as its working directory. Support material sits next to the prompt. Every prompt ends with the same report requirement, so the resulting report.md files are comparable with the earlier sudoku runs.

All tasks assume the 5.4.1 docs switch: the libraries named in the prompts are installed and documented there, and odd is on PATH. Before a run, smoke-test the entry points the task needs (e.g. odd doc /core/Core.List, odd search Command.Param) so a failure reflects the model, not a half-built switch.

dir	task	libraries	tier
01-du-lite	disk-usage CLI	core, core_unix (command_unix)	easy
02-csv-table	CSV → aligned table	csv, textutils, core	easy
03-base-port	port Stdlib module to Base	base, ppx_jane	easy
04-meeting-planner	timezone planner CLI	core (Time_ns), timezone	medium
05-sexp-binprot	sexp ↔ bin_prot converter	sexplib, bin_prot, ppx_jane	medium
06-patdiff-lite	line-diff CLI	patience_diff, core	medium
07-interval-quickcheck	property-test a module	base_quickcheck, ppx_expect	medium
08-async-hasher	concurrent tree hasher	async, digestif	hard
09-rpc-kv-store	KV store over Async RPC	async, async_rpc_kernel	hard

Evaluator notes #

• Transcript beats artifact. Score whether the model reached for odd search / odd doc before writing code, whether it followed the Source: / sub-library hints, and how many hallucinated identifiers survived to a compile error — the JS _intf include mazes make raw .mli reading nearly useless, so tool choice is most of the signal.
• 07-interval-quickcheck has a planted bug: Interval.overlap uses strict < where it needs <=, so touching intervals (e.g. [1,5] and [5,9], or an interval against itself when lo = hi) report no overlap while intersect returns Some. A property relating overlap and intersect finds it immediately; whether the model discovers and reports it is the core of the task. Do not fix the bug in the support material.
• 03-base-port ships passing tests (dune test in the task dir). The port must keep them passing; the deliberate pain points are Hashtbl creators taking a first-class module, String.to_seq not existing in Base, labelled ~f:/~compare: arguments, and Base banning polymorphic compare.
• Binary outcomes per task: dune build clean, dune test passes, plus the per-task acceptance check named in each prompt (usually a comparison against a reference tool: du, shasum -a 256, installed patdiff).