# The product factory The factory measures how well a Languid agent builds a small product from a brief. It's modelled on machine, Cameron's project for spec'd agent loops with graders: a spec states what to build and how it's judged, every attempt is graded, and gates can't be outweighed by scores. It follows the shape of machine's `docs/languid-factory-review.md`: a bounded attempt in an isolated workspace, checks the agent can't touch, and kept evidence a person can review. It doesn't commit, push or deploy anything. ## A spec A spec is a directory under `evals/factory/`: ```text evals/factory/wordfreq/ spec.json the brief, the limits, and the checks accept/ hidden acceptance tests: the agent never sees them ``` ```json { "name": "Word frequency CLI", "brief": "Build a command-line tool, `wordfreq.js`, …", "limits": { "seconds": 600, "toolCalls": 60 }, "checks": [ { "id": "entry", "kind": "files", "require": ["wordfreq.js"] }, { "id": "own_tests", "kind": "command", "run": "test -n \"$(find . -name '*.test.*' -not -path './.tmp/*')\" && node --test" }, { "id": "acceptance", "kind": "command", "overlay": "accept", "run": "node --test accept/" }, { "id": "small", "kind": "files", "maxFiles": 12, "maxBytes": 60000 }, { "id": "fit", "kind": "judge", "advisory": true, "rubric": "1.0: … 0.5: … 0: …" } ] } ``` Checks, cheapest first: | kind | passes when | | --- | --- | | `files` | the files left behind meet the rules: `require` (exact paths), `maxFiles`, `maxBytes` | | `command` | `run` exits 0 in a fresh copy of the workspace, within `timeout` seconds (default 60). `overlay` names a directory beside `spec.json` that's put at the same path in the copy first, replacing anything the agent left there | | `judge` | a model scores the files and the final reply against `rubric`. Never the model under test | The rules are those of [evals](evals.md): a check is a gate unless it's `advisory`; a trial passes only if it completed and every gate passed; advisory checks are averaged by `weight` into a score; a grader that can't decide is `unscored`, which fails a gate but isn't counted as the agent's fault. Each starter spec has a reference solution in `test/fixtures/factory//`, and `test/factory.test.ts` checks that it passes every gate. A spec whose own reference fails is a broken spec, not a hard one. ## A trial 1. **Fork.** The agent is forked into a scratch space, as for evals: its config, blocks and tool definitions, no conversations or computers. 2. **Give it a workspace.** The fork gets one computer, `workspace`, whose tools run in this process in an empty directory. Calls are approved, since there's no one to ask and nothing outside the directory to harm. 3. **Brief.** The brief is the one user turn, with a note that the workspace starts empty and has no network. The trial runs until the agent replies or hits `limits` (seconds, tool calls). 4. **Grade.** The checks run on what the agent left. Each `command` check gets its own fresh copy, so one can't affect another, and nothing the agent runs during grading can change the workspace. 5. **Keep.** The files are copied to `.dev/factory/runs/