README.md

Eval #

Retrieval quality is measured with a known-answer query set: each entry is a plain-language query plus the repository and file that should come back for it. score.py embeds every query, searches the live index, and reports how often the expected file is the top hit and how often it is in the top five.

uv run python eval/score.py                  # unscoped
uv run python eval/score.py --repo-scoped    # each query filtered to its own repo

Run it without CODE_INDEX_DATA set, so it scores the index searches actually use rather than an empty one built nowhere.

Where the query set comes from #

A query set is only worth anything if its answers are real files in real repositories, which makes it a description of whatever tree it was written against. So it is a local, untracked input:

CODE_INDEX_EVAL_QUERIES path to the query set
default eval/queries.local.jsonl, which is gitignored
fallback eval/queries.example.jsonl, committed

The default sits inside the checkout, which is the easy place to find it but also means each git worktree has its own. A set meant to outlive any one checkout belongs somewhere stable — ~/.config/code-index/queries.jsonl — with CODE_INDEX_EVAL_QUERIES pointing at it.

queries.example.jsonl uses this repository as its own corpus, so a fresh clone can run the scorer with nothing set up. It is small and easy, and its numbers are not comparable to anything recorded below — write a set against your own repositories to get a figure worth tracking.

One line of JSON per query:

{"query": "plain language description of what the code does",
 "repo": "directory-name", "path": "src/thing.ts", "symbol": "optional"}

repo is matched against the repository directory's base name and path against the file's path within it. symbol is documentation only; nothing scores on it.

Recorded runs #

  • baseline.md — the first full-corpus hit rates, and the per-repository ranking experiment that was tried and rejected.
  • model-comparison.md — nomic-embed-text-v1.5 against Qwen3-Embedding-0.6B on a nine-repository subset. The reason the default model is what it is.
  • batch-size.md — resident memory and throughput against BATCH_SIZE, with batch-size-sweep.py.
  • refresh-cost.md — what an incremental rebuild costs, with refresh-cost.py.

The corpus for those runs is private, so they describe a repository by language and kind rather than by name, and the hit rates in them cannot be reproduced from this repository alone.

--exclude-path #

A query file that lives inside an indexed repository contains every query's own text verbatim, so it wins on text similarity no matter how good retrieval is. --exclude-path eval/queries drops those hits before ranking. Keeping the query set outside the indexed tree avoids the problem outright; the flag remains for scoring an index built while it was still inside one.