atevals #

outcome evals for agents that use AT Protocol MCP servers like pdsx and the atproto docs MCP. an isolated pi agent gets a task, the evaluator checks the live network itself, and the agent passes only if the network agrees with its answer.

install #

needs uv, an authenticated pi, and a checkout of pi-mcp-adapter (pass its index.ts with --adapter).

uv sync
uv run atevals --help

use #

uv run atevals core --pdsx 0.1.11 --skills --with-docs --trials 3

core runs five tasks that cover what atproto's own writing treats as central: identity, provenance, census, write and history. the other tasks are read and lifecycle (record CRUD against one PDS) and docs (questions graded on atproto.com citations). tasks that write need an authorized test account: --handle, --pds, and EVAL_APP_PASSWORD in the environment.

each run writes a result file plus the agent and judge traces under runs/.

design #

• grade the outcome, not the tool calls — prompts state a goal, and any path that reaches it passes, so the evals compare servers instead of rewarding one API shape.
• ground truth shares no code with the server under test — the evaluator does its own DoH lookups, DID document fetches, sync.getRecord proof checks, relay enumeration, PLC audit log reads and Lexicon resolution.
• objective checks gate the judge — the tool-free judge only scores free-text reasoning, and a failed check can't be overridden by a positive judgment.
• one fresh, sealed agent per phase — only the configured MCP servers are loaded, credentials stay out of the agent's environment, and secrets are redacted from saved traces.
• expected answers are recomputed at run time — fixtures are live accounts, so drift changes how hard a task is, never whether the grade is correct.

develop #

uv sync                                          # install the package and deps
uv run python -m unittest discover -s tests      # run the checks' unit tests
uv run atevals core --cases identity             # run one live task

docs #

docs/ holds the task design and the results of each recorded run.