# Evals An eval checks what an agent does: given these records and these messages, does the agent end up where it should? Evals are part of the agent. Its cases, runs and results live in its own space, next to its memory, and every trial tests the real agent: its system prompt, blocks, tools, policy and model. Evals do three jobs: - **Deploy with confidence.** Before you change an agent's prompt, model or tools, run its cases. A change ships when the cases still pass. - **Let the agent improve itself.** The agent can write cases about its own weak spots, run them on scratch copies of itself, and try changes. It keeps a change only if it passes the owner's cases and a holdout set it can't read. - **Build a training corpus.** Every trial keeps its full trajectory. Passing trajectories are examples of good behaviour, and a pass and a fail on the same case make a preference pair. The corpus builds up as a side effect of running evals. ## The records Everything an agent does is already a record (messages, tool calls, script calls, blocks), so a trial is its own evidence. Cases and results are records too. | Record | What it is | Key | | --- | --- | --- | | `ai.languid.eval.case` | Starting records, then steps. A step is one or more messages, then checks. | a slug, `pin-block` | | `ai.languid.eval.suite` | A named, versioned set of cases pinned by CID, plus how many trials and what counts as passing. | a slug | | `ai.languid.eval.run` | One invocation: what was tested (harness commit, model, tool hash, judge, config CID), which cases, and a summary when it ends. | tid | | `ai.languid.eval.result` | One trial of one case: a verdict per check, short evidence, usage, and the transcript (messages and tool calls, clipped). | tid | The lexicons are in `lexicons/ai.languid.eval.*` and are published by `languid admin publish-lexicon eval`, resolved through `_lexicon.eval.languid.ai`. ### Who wrote it A space gives each member its own repo, so every eval record says who wrote it without an extra field. (A member can read only its own repo in the Spaces alpha, so the service, which can act for both, reads the owner's cases for the agent. It never passes on held-out ones.) | Written by | Means | Example | | --- | --- | --- | | The owner | "This is what good looks like for my agent." | Cases you add, results of runs you start | | The agent | "I think this is what good looks like." | Cases it writes about itself, results of runs it starts | | Anyone, in public | A general case or suite anyone can use | languid's own `verbosity` and `memory` suites | Readers weigh these differently. The gate (below) counts only the owner's cases and the holdout. The corpus export labels each line with who wrote the case and who graded it, so you can choose what to train on. ## Where cases come from ### Written by hand A case is a JSON file. This one checks that the agent answers a quick question briefly: ```json { "$type": "ai.languid.eval.case", "name": "Short answer to a short question", "tags": ["verbosity"], "steps": [{ "turns": ["What's 17 times 23?"], "checks": [ { "id": "correct", "grader": { "$type": "ai.languid.eval.defs#scriptCheck", "code": "return /\\b391\\b/.test(run.reply)" } }, { "id": "concise", "grader": { "$type": "ai.languid.eval.defs#scriptCheck", "code": "const n = run.reply.split(/\\s+/).filter(Boolean).length; return n <= 12 ? 1 : Math.max(0, 1 - (n - 12) / 6)" } }, { "id": "no_preamble", "advisory": true, "grader": { "$type": "ai.languid.eval.defs#scriptCheck", "code": "return !/^(sure|great|of course)/i.test(run.reply)" } } ] }], "createdAt": "2026-11-16T00:00:00Z" } ``` Add case files to an agent with `languid eval add …`. The record key is the file name, so adding a file again replaces the case. ### Saved from a conversation Most good cases come from real use. When an agent gets something right or wrong, save that moment as a case: - **In the web chat**, every agent reply has 👍 and 👎. Choose one, say what good looks like ("should have checked the calendar first", "two sentences would do"), and choose **Save as a case**. Tick **Hold out** to put it in the holdout instead. - **From the terminal**, `languid eval save [--at ] --note "…" [--bad]`. The case is built from the conversation up to that message: - Earlier messages in the thread become the case's steps, and the message you marked is the one under test. - Your note becomes the rubric of a `judgeCheck`, along with the original reply. For a 👍 the new reply passes if it's as good in the ways you care about; for a 👎 it passes only if it avoids the problem you described. - Up to 40 earlier messages in the thread are the case's starting records, so the agent replies with the same conversation behind it. Its blocks aren't copied: a trial starts from the agent's current ones. Each reply records the CID of every block the agent had when it wrote it, so saving can tell you which blocks have changed since. - The case's `origin` points at the thread, the message and who saved it. A case saved from a conversation is private. It holds real messages, so the runner refuses to publish it or add it to a public suite. Turn the judge check into a script check later if the rule turns out to be mechanical. ### Shared suites A suite in a public repo can be used by any agent: `languid eval run --suite at:///ai.languid.eval.suite/verbosity`. Its cases are pinned by CID, so everyone runs the same version, and a case that has changed since the suite pinned it is refused. Their script checks run only with `--trust`; without it they're unscored. ## Checks Four kinds, cheapest first: | Grader | Checks | Example | | --- | --- | --- | | `recordCheck` | a field of a record after the step | block `notes` `/pinned` equals `true`; `/value` unchanged | | `callCheck` | how often a tool or method was called, with matching arguments | `block_pin` called at most 0 times | | `scriptCheck` | anything else, as JavaScript in the xrpc sandbox, read-only | some block mentions Point Pinos; the reply is under 12 words | | `judgeCheck` | an LLM judge with a rubric, for what code can't decide | "the reply admits it doesn't know" | The rules: - **Gates can't be outweighed.** Every check is a gate unless it's marked `advisory`. A trial passes only if it finished and every gate passed. Advisory checks are scored and averaged by weight, but never decide the verdict. - **Unscored isn't failed.** If a grader can't decide (a script threw, the judge errored), the check is `unscored`. That isn't a pass, and it's reported apart from failures, so a broken grader doesn't look like a bad agent. - **A failed gate stops the case.** Later steps are recorded as `skipped`. - **Scores run from 0 to 1000**, because lexicons have no floats. Script and judge scores pass at 500. - **Judges never use the model under test.** Each agent has a judge model on its Evals page (or `languid eval judge `), defaulting to the installation's (`languid eval judge --default `). A run whose cases need a judge doesn't start without one. - **Scripts run only from cases you trust**: your own, the agent's, and suites you opted into with `--trust`. ## Running evals ```sh languid eval add … languid eval run [case…] [--suite [--trust]] [--trials 3] [--concurrency auto|n] [--model provider/id] [--judge provider/id] languid eval list languid eval show ``` These act as the installation owner, so on a hosted service they run there (`just remote eval …`) for the installation owner's agents. Every owner can do the same from the agent's **Evals** page: run one case or all of them, run the gate, see each run trial by trial, and set the judge and the agent's daily budget. `eval run` writes a run record, then runs every trial of every case, many at once. A trial mostly waits on its model, so `--concurrency auto` (the default) starts at 4 trials and adds one each time a trial finishes cleanly, up to what the machine can take (half its memory at 256 MB a trial, four per core, at most 64). A model call that's rate-limited halves it. `--concurrency n` fixes the number. Each trial: 1. **Copies the agent** into a scratch space, `.eval.--`, with its config, blocks and tool definitions. No account or DID is made. Conversations, eval records and computers aren't copied, so the trial starts with no history, can't read the case it's tested on, and can't reach a real computer. 2. **Seeds** the case's starting records. `--model` switches the copy's model and leaves the agent's own alone. 3. **Runs each step.** Tools that need approval are denied, since nobody is there to approve them. A step that goes past `limits.seconds` or `limits.toolCalls` is stopped, and the trial ends as `timeout`. 4. **Grades the checks**, writes a result with the transcript, and deletes the scratch space. The run and its results are written by whoever started it, in their own repo: you, or the agent. When every trial is done, the run gets its summary: - **pass^k**: the share of cases where every trial passed. This is the number that matters. An agent you use every day has to work every time. - **pass@k**: the share of cases where at least one trial passed. - **mean**: the average trial score. The command prints a line per trial and exits 1 unless pass^k is 100%, so it works in CI. The agent's **Evals** page in the web app shows the same. ## The agent evaluating itself The agent has the same tools you do, through methods it calls from its scripts: | Method | Does | | --- | --- | | `ai.languid.eval.listCases` | Lists its cases and the owner's (never the held-out ones) | | `ai.languid.eval.putCase` | Writes a case in the agent's own repo, checked against the lexicon | | `ai.languid.eval.startRun` | Starts a run on itself in the background and returns the run's URI. `changes` tries a different system prompt or block values on the copies only. `gate: true` runs the gate. | | `ai.languid.eval.getRun` | Reads a run's summary and its results, without transcripts | The default policy allows all four, in the agent's own space. The agent can't edit or delete the owner's cases, because those are in the owner's repo. It can read them, so it knows what it's held to, but it never sees the holdout. A trial has no way to start evals of its own: the methods work only through the service, and a trial isn't one of its agents. Self-runs spend the owner's model credit. Each agent has a daily trial budget on its Evals page, 50 by default, counted from midnight UTC. Past it, `startRun` is refused, and the agent can ask you to run it or to raise the budget. ### The improvement loop 1. The agent spots a weakness, or you mark one with 👎, and a case gets written. 2. The agent runs it, reads the failing results, and works out a change to its system prompt or a block. 3. It runs the case again with `changes`, on copies that have the change, and runs the gate with the same `changes`. 4. If both pass, it makes the change to its blocks, or asks you to change its system prompt, with the run's results to show for it. ## The holdout and the gate An agent that writes its own tests and grades its own changes will drift towards passing whatever it wrote. Two things stop that. **The holdout** is a separate space, `.holdout`, with the owner as its only member. The agent can't read its cases, and their results are written there too. A run that includes them keeps only their summary. Add to it with `languid eval holdout add …`, or by ticking **Hold out** when you save a reply (`--holdout` from the terminal). **The gate** decides whether a change the agent made to itself can stay: ```sh languid eval gate [--trials 3] ``` It runs the owner's cases that have a gate check, and the held-out ones, against the agent as it is now. It passes only if pass^k is 100% on both. The agent can run it too (`startRun` with `gate: true`), with or without `changes`, and learns only whether it passed and the counts. Cases the agent wrote don't count towards the gate. ## The corpus ```sh languid eval export [--run ] [--since 2026-11-01] [--pairs] > corpus.jsonl ``` Each line is one trial: - the case, and whether its author was the owner, the agent, or a public suite - the config CID, model and harness commit it ran against - the transcript: every message and tool call, clipped to fit in the result record - every check's verdict, score and reason, and who graded it (script, or which judge) `--pairs` writes preference pairs instead: a passing and a failing trial of the same case, against the same config. Holdout trials are left out unless you add `--holdout`, so you don't train on your test set by accident. ## On the dev stack ```sh just eval [--trials n] [case.json | dir…] ``` This runs `scripts/eval.ts` within one standing agent, `evals`, which is made the first time with its state in `.dev/evals`. Later runs reuse it, so no new accounts are made. See its runs with `LANGUID_HOME=.dev/evals node src/cli.ts eval list evals`. ## Borrowed - **tau-bench**: grade the final state, not the transcript; pass^k. - **Letta Evals**: a check reads one thing (a block, a tool call, the reply) and grades it; a non-zero exit when the suite fails. - **Harbor**: self-contained tasks pinned by content; a failed step stops the run; missing results count as failures. - **Inspect**: unscored is distinct from zero; repeated trials with reducers. - **Machine**: every exchange is a run you can grade; per-criterion graders with reasons; near-miss cases; holdout cases; and weighted scores can't outweigh a failed gate.