this post sux
sux bench
1 folder · 32 files

data
.gitignore
analyze.py
counters.py
embed_pool.mjs
escalation.mjs
fetch_posts.mjs
fuzzy.mjs
make_set.mjs
make_tasks.mjs
package-lock.json
package.json
pos_only.py
pyproject.toml
README.md
regress_baseline.json
regress.md
regress.mjs
results_counters.md
results_escalation.md
results_fuzzy.md
results_pos_only.md
results_train_from_jev.md
results_wording.md
results_wording2.md
results.md
run_gemma.mjs
run_jev.mjs
run_laya.py
train_from_jev.mjs
uv.lock
wording.mjs
wording2.mjs
README.md

bench: Jev vs Laya vs EmbeddingGemma for in-browser feed filtering #

100 current Bluesky Discover posts, labeled by Claude on five sux-style rules, asked with the exact question wording sux sends Jev (data/tasks.json is generated from shared.js). Results: results.md.

node fetch_posts.mjs 300          # data/pool.json (skips !no-unauthenticated authors)
# label data/pool_part{0,1,2}.json -> data/labels_part{0,1,2}.json (done by Claude subagents)
node make_set.mjs                 # data/set.json: 100 posts, rare positives kept
node make_tasks.mjs               # data/tasks.json from sux's buildQuestions
source ../typesafe_api_key.secret.sh && node run_jev.mjs
uv run python run_laya.py english mps     # also: multilingual, cpu
node run_gemma.mjs q8                     # also: fp32, q4 (Transformers.js, onnx-community build)
uv run python analyze.py                  # -> results.md

Caveats: one labeler family (Claude), 100 posts, 12–31 positives per rule, so differences under ~0.05 are noise. Gemma latency is Node/onnxruntime CPU, not browser WASM/WebGPU. Laya ships no ONNX or npm package; its timings are PyTorch.