Eval #
Retrieval quality is measured with a known-answer query set: each entry is a
plain-language query plus the repository and file that should come back for it.
score.py embeds every query, searches the live index, and reports how often the
expected file is the top hit and how often it is in the top five.
uv run python eval/score.py # unscoped
uv run python eval/score.py --repo-scoped # each query filtered to its own repo
Run it without CODE_INDEX_DATA set, so it scores the index searches actually
use rather than an empty one built nowhere.
Where the query set comes from #
A query set is only worth anything if its answers are real files in real repositories, which makes it a description of whatever tree it was written against. So it is a local, untracked input:
CODE_INDEX_EVAL_QUERIES |
path to the query set |
| default | eval/queries.local.jsonl, which is gitignored |
| fallback | eval/queries.example.jsonl, committed |
The default sits inside the checkout, which is the easy place to find it but also
means each git worktree has its own. A set meant to outlive any one checkout
belongs somewhere stable — ~/.config/code-index/queries.jsonl — with
CODE_INDEX_EVAL_QUERIES pointing at it.
queries.example.jsonl uses this repository as its own corpus, so a fresh clone
can run the scorer with nothing set up. It is small and easy, and its numbers are
not comparable to anything recorded below — write a set against your own
repositories to get a figure worth tracking.
One line of JSON per query:
{"query": "plain language description of what the code does",
"repo": "directory-name", "path": "src/thing.ts", "symbol": "optional"}
repo is matched against the repository directory's base name and path against
the file's path within it. symbol is documentation only; nothing scores on it.
Recorded runs #
baseline.md— the first full-corpus hit rates, and the per-repository ranking experiment that was tried and rejected.model-comparison.md— nomic-embed-text-v1.5 against Qwen3-Embedding-0.6B on a nine-repository subset. The reason the default model is what it is.batch-size.md— resident memory and throughput againstBATCH_SIZE, withbatch-size-sweep.py.refresh-cost.md— what an incremental rebuild costs, withrefresh-cost.py.
The corpus for those runs is private, so they describe a repository by language and kind rather than by name, and the hit rates in them cannot be reproduced from this repository alone.
--exclude-path #
A query file that lives inside an indexed repository contains every query's own
text verbatim, so it wins on text similarity no matter how good retrieval is.
--exclude-path eval/queries drops those hits before ranking. Keeping the query
set outside the indexed tree avoids the problem outright; the flag remains for
scoring an index built while it was still inside one.