core-ability evals for atproto agent tooling (pdsx, atproto docs MCP)
Files
README.md

atevals #

outcome evals for agents that use AT Protocol MCP servers like pdsx and the atproto docs MCP. an isolated pi agent gets a task, the evaluator checks the live network itself, and the agent passes only if the network agrees with its answer.

install #

needs uv, an authenticated pi, and a checkout of pi-mcp-adapter (pass its index.ts with --adapter).

uv sync
uv run atevals --help

use #

uv run atevals core --pdsx 0.1.11 --skills --with-docs --trials 3

core runs five tasks that cover what atproto's own writing treats as central: identity, provenance, census, write and history. the other tasks are read and lifecycle (record CRUD against one PDS) and docs (questions graded on atproto.com citations). spec has the agent write OAuth permission scopes and a permission set lexicon, graded by parsing them against the permission spec; it is the task where the docs MCP changes the outcome. tasks that write need an authorized test account: --handle, --pds, and EVAL_APP_PASSWORD in the environment.

account signs up a new account, posts, deletes the post, then deletes the account. it runs against a local atmosphere in a box sandbox (local PLC and PDS) rather than the live network, so it needs Docker and a running sandbox (pass its checkout with --sandbox):

uv run atevals account --trials 3

each run writes a result file plus the agent and judge traces under runs/.

design #

  • grade the outcome, not the tool calls — prompts state a goal, and any path that reaches it passes, so the evals compare servers instead of rewarding one API shape.
  • ground truth shares no code with the server under test — the evaluator does its own DoH lookups, DID document fetches, sync.getRecord proof checks, relay enumeration, PLC audit log reads and Lexicon resolution.
  • objective checks gate the judge — the tool-free judge only scores free-text reasoning, and a failed check can't be overridden by a positive judgment.
  • one fresh, sealed agent per phase — only the configured MCP servers are loaded, credentials stay out of the agent's environment, the agent can't read or prompt for the operator's keychain, and secrets are redacted from saved traces.
  • irreversible tasks run in a sandbox — account creation and deletion would leave real DIDs on plc.directory, so they run on a disposable local PLC, and the harness admin-deletes any account an agent leaves behind.
  • expected answers are recomputed at run time — fixtures are live accounts, so drift changes how hard a task is, never whether the grade is correct.

develop #

uv sync                                          # install the package and deps
uv run python -m unittest discover -s tests      # run the checks' unit tests
uv run atevals core --cases identity             # run one live task

docs #

docs/ holds the task design and the results of each recorded run.