diff --git a/01-wrong-turns-and-traps.md b/01-wrong-turns-and-traps.md index decb7d4..d07633a 100644 --- a/01-wrong-turns-and-traps.md +++ b/01-wrong-turns-and-traps.md @@ -201,6 +201,19 @@ 40. **Measured End-to-End Speed (`32K` Context, `6.5 GB` Arena, Unpruned `K=512` `IQ2_XS`, `call_3817687`, `call_2483052`, `call_3102851`, `call_2898988`, `call_4110683`, `call_4178375`, `call_3948986`, `call_4500449`, `call_3444931`, `call_3116575`, `call_4373130`, `call_2769318`, `call_3781775`, `call_4410602`, `call_3511955`, `call_4432071`, `call_2918543`, `call_1514614`, `call_1173886`, `call_2456120`, `call_5024768`, `call_4223761`, `call_3543222`, `call_1624568`, `call_3678353`, `call_4256391`, `call_3418394` & `call_3706633`):** `prompt_2048tok_ops` prefill **`400.88–432.78 tok/s` (`4,584.29–4,949.14 ms` end-to-end — all-time record `4,584.29 ms` / `432.78 tok/s` — `-1,728 ms` / `+37.7%` faster end-to-end, **`3,131.29 ms` / `633.28 tok/s` pure GPU `kprof` — `3,267.43 ms` serialized sum!**, `2,047` dispatches (`-47` dispatches/chunk!), `1.12 GB` transient scratch!)**, decode **`41.53–42.27 tok/s`** (`23.66–24.08 ms/tok`, **`3,004.68–3,005.02 ms` decode, `mtp_ms = 190.62–197.99 ms` / `1.63 ms/draft` — new all-time MTP record `190.62 ms`!**); `prompt_albedo_11k` (`10,180` tokens) prefill **`421.49–446.59 tok/s` (`22.79–24.15 s` vs `28.25 s`, `-5.46 s` faster — all-time record; Chunk 0 = `4,613–4,719 ms`, Chunk 1 = `501.5 tok/s` / `4,084–4,572 ms`, Chunk 2 = `4,362–4,736 ms`, Chunk 3 = `4,610–4,911 ms`, Chunk 4 = `4,685–4,898 ms`, `mtp_ms = 121.54–125.14 ms`, and `@10.2K` decode hit **`25.00–25.26 tok/s` / `39.59–40.01 ms/tok` / `1,227.18–1,240.21 ms` — all-time record `25.26 tok/s` at `10.2K` context!** vs `22.06 tok/s` / `1,405.37 ms`!)**; `coding` prefill **`120.61–126.78 tok/s` (`2,413–2,537 ms`, `+32.2%`)**, decode **`35.46–37.1 tok/s`** (**`6,115.57–6,176.28 ms` decode — all-time record `6,115.57 ms`, `mtp_ms = 374.22–380.29 ms`**); `reasoning` prefill **`121.57–122.92 tok/s` (`2,416–2,458 ms`, `+33.3%`)**, decode **`34.92–36.74 tok/s`** (**`5,961.16–6,270.75 ms` decode, `mtp_ms = 322.48–370.78 ms` — all-time record `322.48 ms`!**); `persona_chat` prefill **`124.62–128.14 tok/s` (`2,255.32–2,443 ms`, `+38.5%` — all-time record `2,255.32 ms` / `128.14 tok/s`!)**, decode **`28.50–33.2 tok/s`** (**`6,628.00–7,685.17 ms` decode, `mtp_ms = 374.58–424.81 ms` — all-time record `374.58 ms`!**). + +42. **recurrent-state INT4 / stochastic-rounding quantization in Gated DeltaNet** ([2610.15139](https://arxiv.org/abs/2610.15139), [2610.15709](https://arxiv.org/abs/2610.15709)): + - *trap:* treating recurrent state like a once-written KV cache. with per-token dynamic scaling, $s_{t,c}=\max|\widehat S_{t,c}|/q_{\max}$ reads its own quantization-corrupted state: outliers increase scale, bulk resolution coarsens, and subsequent errors feed back into the state. scale-yoking to the FP trajectory removes numerical runaway but leaves substantial functional damage; bounded norm is not preserved language-model quality (`2610.15139`, `sec_mechanism.tex`, `sec_whatworks.tex`). + - *source corrections:* stochastic rounding raises per-step RMS error by $\sqrt{2}$ (measured $1.42\times$), **not $\sqrt{23}$**; on GDN-0.8B at yoked INT4 it adds `6.71` nats excess NLL. the **first GDN layer alone** accounts for `63–78%` of summed single-layer INT4 damage at `H=2048`; this is not a measured first-three-layer error fraction. protecting the first three recurrent layers is a separate matched-budget allocation result, identical to rescue selection/knapsack on their small GDN models, not established for Swift. + - *mechanism beyond runaway:* small updates vanish on the coarse state grid and errors can accumulate coherently: $E_T=\sum_{\tau\le T}A_{\tau:T}R_\tau$, with off-diagonal error inner products in $\|E_T\|_F^2$. `2610.15709` preserves updates with a low-rank bypass (LRB) and bounded directional rounding (BDR), rather than assuming unbiased noise is safe. its GDN result uses `5.25` average / `6.00` peak state bits including bypass, not bare INT4; WikiText-103 8K PPL is `26.85` vs FP32 `24.19` and naive INT4 `79.76`. + - *rule:* keep Swift's existing high-precision recurrent path. our roughly `112.2 MiB` fixed GDN state is small next to the `6.50 GB` expert arena; do not trade it for unvalidated long-horizon drift. any future state compression must test natural-continuation KLD/NLL, long exposure, session/SCR and speculative rollback, not just NIAH or state norms. LRB/BDR and ReplaySSM are research candidates, not proof that arbitrary recurrent INT4 is safe. + +43. **routing-frequency / entropy / phase-fragility proxy bit allocation without measured distortion curves** ([2610.13328](https://arxiv.org/abs/2610.13328), [2610.20826](https://arxiv.org/abs/2610.20826)): + - *trap:* using a cache-popularity rank, routing entropy, or phase-specific fragility as a substitute for $d_e(M)$ (Hessian-/activation-/decision-weighted error or direct ablation). DuoRate Proposition 1 cancels frequency only when the **same positive** $f_e$ weights distortion and read cost: $\arg\min_M f_e[d_e(M)+\lambda R(M)]=\arg\min_M[d_e(M)+\lambda R(M)]$. promotion is ordered by $\Delta d_e/\Delta R$, not popularity. Proposition 2 retains frequency under an unweighted storage constraint and through cross-domain mixtures $\pi_{d\mid e}=w_df_{d,e}/\sum_{d'}w_{d'}f_{d',e}$; it does **not** say frequency is useless for cache admission or SSD traffic. + - *negative result:* on **Qwen-3.6-35B-A3B**, `2610.20826`'s phase/entropy/CV/Jaccard/top-1 proxy is neutral near 4 bpw and loses at the steep IQ2 floor: uniform KLD `0.034321`, proxy `0.038142–0.038249`, random `0.037728/0.037808` (random marginally better, within about 1 SE of proxy). proxy p99 KLD rises `0.437 -> 0.487–0.492`; mandatory downgrades inject tail damage. this is a coding-domain, layerwise negative result, not a theorem against all mixed precision or a measurement of our IQ2 formats. + - *our boundary:* the K512 header has **144 packed expert tensors**, not 144 independently precision-assigned experts: `68 IQ2_S`, `48 Q2_0`, `24 IQ2_XXS`, `4 IQ3_S`; each tensor packs 512 experts with one type. ukisai's imatrix/GSQ-RCO artifact is not DuoRate's HQQ/GPTQ/MiLo/MoEQuant/BitsMoE baseline. its IQ codes are not prefix-readable StagQ planes; truncating their bytes is corruption, not Stage-2 read-tier selection. + - *rule:* preserve the current quant assignment unless a format-/byte-/arena-matched ablation beats it on mean **and tail** KLD, top-1, persona/JA/ops/tool behavior and measured SSD/GPU time. cache ranks may optimize reuse, but must not silently become precision ranks. cache-conditioned SSD miss weights also differ from distortion weights, so Proposition 1 cannot be copied unchanged into `stream.cpp`. + ### B. ruled out from the original blueprint details and primary-source checks in `blueprint_review.md`. - **the blueprint's expert size math and mixed-quant expert tiers:** claimed 224 experts × 48 layers at Q3_K is 14.2 GB (real: **22.7 GB**, or 16.5 GB at Q2_0), and proposed mixing Q3_K hot experts with Q2_K cold experts inside one layer (impossible in GGUF: `ffn_*_exps` is one 3D tensor per layer with one quant type [src: ISTA card]). diff --git a/04-sources-and-references.md b/04-sources-and-references.md index dd38c15..48bafed 100644 --- a/04-sources-and-references.md +++ b/04-sources-and-references.md @@ -55,6 +55,43 @@ - 2602.02099: DDCA · 2505.14604: SBT · 2505.20501: Gatsby lipogram (misattributed by the blueprint) - 2407.11062: EfficientQAT (blueprint background) + +### october 2026 arXiv preprints — implications for `swift-1.5-k512-iq2xs` + +primary evidence: extracted LaTeX in `/tmp/arxiv_src//`; these are preprints, not independently reproduced Swift results. equations below retain the papers' conventions; deployment adaptations are marked explicitly. + +- **2610.22454 — *Where Prefill Expert Reduction Hurts MoE Models*.** https://arxiv.org/abs/2610.22454 · https://arxiv.org/pdf/2610.22454 + - `main.tex`, background and §§“What Is Lost, and Where” / “Where the Full-Expert Budget Goes”: a reduced token retains its top $k'\epsilon_B\Delta_t$, with an in-range-grid guard; overshoot is bounded by $|d|/\epsilon_B$. + - GDN configuration: rank `1`, merge interval `15`, $\epsilon_L=0.50$, $\epsilon_B=0.30$, `5.25` average / `6.00` peak bits including bypass. WikiText-103 8K PPL `24.19` FP32 / `79.76` INT4 / `26.85` LRB+BDR. for Swift: investigate only if state traffic becomes material; published 1.3B, unquantized-weight, RTX 3060 batch `16/32/48` evidence does not validate this hybrid GSQ-RCO model. it complements the scale-feedback warning; it is not evidence for naive INT4 or stochastic rounding. + ### hardware - RX 7600 XT: https://www.techpowerup.com/gpu-specs/radeon-rx-7600-xt.c4190 - Apple M5: https://support.apple.com/en-us/125405