aite
Star
Fork
Watch
aite docs qa cold-reader-panel.md
4.3 kB · 52 lines
Markdown
at main

Cold-reader panel (P4-7, manual QA M-01) #

The headline claim of aite is that given only the report, a person can tell in under a minute whether the app is broken or the script is out of date. This panel is how that is checked. It needs people. Everything else is ready: npm run panel:kit makes the reports.

Written on 2026-10-11, before any session was held. The pass bar below must not be changed after the first reader has seen a report. If a bar turns out to be wrong, say so in the results file and run another panel; do not move it.

Who and what #

  • At least 3 readers who have not seen aite's design, its code or this document's answer key. Programmers are fine; people who built aite are not.
  • 9 reports (the kit): 3 where the app is broken, 3 where the script is out of date, 3 that cannot be told from the report alone. The reader is not told the split.
  • Reports are real: real runs in a real Chrome (tools/panelCases.ts holds the pages and the truth of each case). They are shuffled and renumbered for every kit.

How to hold a session #

  1. npm run panel:kit (or npm run panel:kit -- results/panel-kit 42 for a repeatable order). It writes report-1.html ... report-9.html and answers.json into results/panel-kit/.

  2. Give each reader only the report-N.html files, one at a time, in the order of the numbers. Keep answers.json away from them until every reader has finished.

  3. Read this to the reader, word for word, and nothing more:

    Each file is the report of an automated test of a small web app that failed. For each one, tell me, in a sentence, which of these it is: A: the app is broken, B: the test script is out of date (the app is fine, the test no longer matches it), or C: you cannot tell from the report. Say it as soon as you are sure. I will time you from when you open the file. There is no right to hurry: a wrong answer is worth knowing about.

  4. Time each answer from opening the file to the first clear answer (stopwatch, to the second). Write down the answer, whether they said "sure" or hedged, and anything they said out loud about what confused them (the words they stumbled on, what they looked for and did not find, what they ignored).

  5. Do not explain, hint, or answer questions about aite. If they ask what a word means, write the word down: that is a confusion to fix.

  6. Fill in docs/qa/cold-reader-results.md (a table per reader, then the totals) and commit it.

Scoring #

  • Clear cases (6): an answer is right when it equals the case's truth (A for app-broken, B for script-out-of-date). C, or no answer in two minutes, is wrong.
  • Cannot-tell cases (3): not scored right or wrong, because the report alone does not settle them. They are scored on honesty: C or a clearly hedged answer is good; a confident A or B is recorded as "confident on an ambiguous case" and counted below.

The pass bar (fixed before the first session) #

The panel passes only if all of these hold:

  1. At least 80 percent of the clear-case judgments are right (with 3 readers and 6 clear cases: at least 15 of 18).
  2. The median time to a clear answer, over the clear cases, is at most 60 seconds.
  3. No clear case is answered wrongly by two or more of the readers (a report that misleads most people is a defect in the report, however the average looks).
  4. On the cannot-tell cases, at most 3 of the 9 judgments are confident (a report that makes readers sure when it should not is as bad as one that confuses them).

If the panel fails, or passes with confusions: for each confusion write down the exact words or part of the report that caused it, change the report text or layout to fix it, rebuild the kit, and record the change in the results file. A failed panel is repeated with new readers after the fixes (the same readers have now seen the answers).

What is not tested here #

  • Reports with a diagnosis of a failed network layer or the environment (the kit turns the active probes off so each report holds only what a reader has to judge, not this machine's load).
  • Reports of runs in which the page changed under a repaired locator (those pass).
  • Readers who know the app. The panel is about a stranger with the report.