The product factory #
The factory measures how well a Languid agent builds a small product from a brief. It's modelled on machine, Cameron's project for spec'd agent loops with graders: a spec states what to build and how it's judged, every attempt is graded, and gates can't be outweighed by scores. It follows the shape of machine's docs/languid-factory-review.md: a bounded attempt in an isolated workspace, checks the agent can't touch, and kept evidence a person can review. It doesn't commit, push or deploy anything.
A spec #
A spec is a directory under evals/factory/:
evals/factory/wordfreq/
spec.json the brief, the limits, and the checks
accept/ hidden acceptance tests: the agent never sees them
{
"name": "Word frequency CLI",
"brief": "Build a command-line tool, `wordfreq.js`, …",
"limits": { "seconds": 600, "toolCalls": 60 },
"checks": [
{ "id": "entry", "kind": "files", "require": ["wordfreq.js"] },
{ "id": "own_tests", "kind": "command", "run": "test -n \"$(find . -name '*.test.*' -not -path './.tmp/*')\" && node --test" },
{ "id": "acceptance", "kind": "command", "overlay": "accept", "run": "node --test accept/" },
{ "id": "small", "kind": "files", "maxFiles": 12, "maxBytes": 60000 },
{ "id": "fit", "kind": "judge", "advisory": true, "rubric": "1.0: … 0.5: … 0: …" }
]
}
Checks, cheapest first:
| kind | passes when |
|---|---|
files |
the files left behind meet the rules: require (exact paths), maxFiles, maxBytes |
command |
run exits 0 in a fresh copy of the workspace, within timeout seconds (default 60). overlay names a directory beside spec.json that's put at the same path in the copy first, replacing anything the agent left there |
judge |
a model scores the files and the final reply against rubric. Never the model under test |
The rules are those of evals: a check is a gate unless it's advisory; a trial passes only if it completed and every gate passed; advisory checks are averaged by weight into a score; a grader that can't decide is unscored, which fails a gate but isn't counted as the agent's fault.
Each starter spec has a reference solution in test/fixtures/factory/<slug>/, and test/factory.test.ts checks that it passes every gate. A spec whose own reference fails is a broken spec, not a hard one.
A trial #
- Fork. The agent is forked into a scratch space, as for evals: its config, blocks and tool definitions, no conversations or computers.
- Give it a workspace. The fork gets one computer,
workspace, whose tools run in this process in an empty directory. Calls are approved, since there's no one to ask and nothing outside the directory to harm. - Brief. The brief is the one user turn, with a note that the workspace starts empty and has no network. The trial runs until the agent replies or hits
limits(seconds, tool calls). - Grade. The checks run on what the agent left. Each
commandcheck gets its own fresh copy, so one can't affect another, and nothing the agent runs during grading can change the workspace. - Keep. The files are copied to
.dev/factory/runs/<time>/<slug>-<trial>/, the fork is deleted, and the report (report.jsonbeside them) records every check, the tokens, the tool calls and the cost.
Isolation #
The agent's commands, and every command check, run confined:
- macOS:
sandbox-exec, with writes allowed only in the workspace (and its jobs directory), no network, and no reads of the home directory. - Linux:
bwrap, with the root read-only, the home directory hidden, the workspace bound writable, and every namespace unshared, so no network. - Either way the environment is replaced (
env -i), so API keys and tokens in the factory's own environment aren't visible.
The file tools (read, write, edit, ls, grep, find) run in the factory's process, so they check paths instead: a path outside the workspace, by .., an absolute path or a symlink, is refused. Without sandbox-exec or bwrap the factory refuses to run.
This is a guard against an agent wandering off, not a boundary for hostile code: the sandbox profile allows everything else by default, and the processes run as you. Don't run specs you didn't write.
Running it #
just factory anthropic/claude-sonnet-5-5 --judge openai-codex/gpt-6-luna # every spec, one trial each
node scripts/factory.ts --model xai/grok-4.7 --trials 3 evals/factory/csv # one spec, three trials
node scripts/factory.ts --model xai/grok-4.7 --trials 5 --concurrency 8 # eight trials at once
Trials run at once, as for evals: --concurrency auto (the default) adapts to the machine and to the provider's rate limits. On the dev stack, the first run makes an owner and an agent named factory, with their state in .dev/factory, and later runs reuse them. --home <dir> --agent <name> runs it on another agent of an installation you own. It prints a line per trial, then pass^k, pass@k, the mean, the total cost and the cost per passing product, and exits 1 unless every trial passed.
Next #
- Results as
ai.languid.eval.runandai.languid.eval.resultrecords in the agent's space, once the evals lexicons settle, so factory runs sit beside the agent's other evals. - Harder specs: a multi-file service, a change to an existing codebase (seed files in the workspace), a bug fix whose regression test must fail before and pass after.
- Patches to Languid itself, which is phase 1 of machine's review: a task contract, a candidate checked out from
origin/main,just checkas the gate, and a person reviewing the patch.