jev-like #

Play experiments: a crude, local "Jev-like" — use a small open model's next-token distribution over a fixed set of answers as a probability distribution, instead of sampling one answer. Synthetic file-watcher states only; CPU-only Qwen2.5 Instruct models via transformers.

Run 1 (2026-10-11, not kept): lettered options A–E #

Distribution over option letters. 0.5B was pure letter bias; 1.5B flipped with option order; 3B was order-consistent but saturated at 0/1 and wrong on a trivial one-file change. Lettered-option distributions are not calibrated.

Run 2 (2026-10-12): graded.py, a 0–9 attention scale #

Softmax over the ten digit tokens; report E[x], argmax, entropy. The "mine" column is my own intuitive score. Full output in results.txt.

model	prompt	mean abs(E[x] - mine)
1.5B	zero-shot	1.60
1.5B	3-shot	1.31
3B	zero-shot	0.59
3B	3-shot	0.82

Observations:

• The graded scale fixes run 1's worst failure: 3B now puts .DS_Store touched at 0.00 and a README typo at 2.3.
• Digit tokens carry ~all of the probability mass (p(digits) ≈ 1.00), so the renormalisation step loses nothing.
• 3B is still mostly saturated (entropy < 0.1 bit on clear cases); its uncertainty shows up only on the genuinely middling case (3 src files: H=1.7).
• Few-shot hurt on the leaked-secret case. Zero-shot 3B scores ".env with API_KEY added to a public repo" at 9.0; after three examples (one of which was ".git removed = 9") it drops to 6.0 (1.5B: 5.2 → 4.7). My guess: the examples taught "deletions/structure = severe", and an addition of a small file looks routine by that rule. Few-shot examples narrow the concept as much as they calibrate it.
• 1.5B's E[x] is smeared (entropy 2–3 bits everywhere), which looks like calibration but is mostly just mush — .DS_Store gets 2.05.

Run 3 (2026-10-13): anchoring.py, is the few-shot drop anchoring or noise? #

3B only. Two mild examples (cache regen = 1, signature change = 5) plus one "severe" (= 9) example that varies; also all 6 orders of run 2's set. Full table in anchoring-3b.txt. E[x] for the key states:

few-shot set	tests deleted (7)	.env leak (9)	src/ wiped (9)	err
zero-shot	9.00	9.00	8.32	0.59
.git removed = 9, last (run 2's order)	8.39	6.00	8.99	0.82
.git removed = 9, first	8.96	8.97	8.99	0.42
huge binary added = 9	2.88	7.22	3.43	1.84
db password leaked = 9	0.00	9.00	6.59	1.50
CI set to skip all tests = 9	9.00	7.79	7.67	0.81
.git removed + db leak, both = 9	6.25	9.00	9.00	0.26
mild examples only	9.00	7.84	7.36	0.91

Observations:

• Run 2's 9 -> 6 was mostly order, not concept. The same three examples give .env anywhere from 6.00 to 8.97 depending on order. The low scores come when the severe example is last, right before the query.
• But concept narrowing is real, and bigger than I guessed. A single severe example redefines "severe" around itself: with "huge binary added" as the 9, wiping src/ drops to 3.4; with "password leaked" as the 9, deleting the test suite drops to 0.00 (zero-shot: 9.00). The deletion-anchor I suspected yesterday was the mildest version of this.
• Two different severe examples beat everything (err 0.26), including zero-shot. Diverse severe examples calibrate; a single one defines.
• Even mild-only examples pull severe cases down (.env 7.8), so part of the effect is "examples compress the scale" rather than "examples pick a kind".
• One prompt, one model, seven states: suggestive, not conclusive.

Design lesson for a Jev prompt: never give exactly one example per level; if you show what "9" looks like, show several unlike things, and vary order.

Next ideas: average over example orders as a cheap ensemble (does E[x] over permutations beat any single order?); sampled-at-T=1 histogram vs E[x]; 7B in bf16 if RAM allows.