Recipe: Bonsai 2 on one RX 7900 XTX #
For humans: this is an agent-written notebook of the optimizations we actually built, measured, kept, and rejected while running Bonsai 2 27B on a 24 GiB RX 7900 XTX reference workstation. Give it to your agent and ask it to work through the recipe against your hardware and workload. It is documentation, not another runtime, installer, collection of weights, or promise that every knob helps.
The format is inspired by astrra.space/ds4-recipe: numbered, agent-facing explanations and a results ledger.
For implementing agents #
Start with 00 — environment, then 01 — correctness and measurement. This recipe documents the deltas on top of a pinned Prism llama.cpp fork, not a reimplementation of the model architecture. Read the public upstream source, model metadata and the mechanisms in these chapters before porting a technique. You do not need the original research notes, Nix files or patch bundle: the lessons are explained here. The full optimized profile still needs an implementation of the documented deltas; an unmodified base build is not it.
This is for running Bonsai generally. Use whatever compatible client fits your purpose: an OpenAI-compatible application, a coding agent, a local chat UI, or direct server requests. Neither FangAgent, Pi, dictation, KDE nor systemd is required to use the runtime techniques. Their names occur only where they identify this workstation, reference packaging, or the consumer we actually used to test a behavior. The runtime patches happen to live in the FangVoice source tree; the recipe is independent of that application's installation.
Our measured reference profile kept 128K context, a normal 32K output allowance, thinking, tools and image support. We did not obtain speed by removing those capabilities or overclocking. Those settings are comparison invariants, not mandatory choices for every Bonsai use case. Quantized KV is a deliberate memory/precision tradeoff; the trained draft head is community work, not an official checkpoint.
Reading order #
| Chapter | What it owns |
|---|---|
| 00 — start here | Hardware, workload, applicability and invariants |
| 01 — correctness and measurement | Oracles, cold/reuse methodology and misleading checks |
| 02 — source, build and artifacts | Immutable inputs, HIP build, patch inventory and deployment proof |
| 03 — weights, memory and KV | PQ2 layout, unchanged target, mixed KV and memory accounting |
| 04 — PQ2 prefill and decode | Unpack sharing, scale staging, tile/row geometry and rejected kernels |
| 05 — MTP, ngram and checkpoints | Trained depth two, reuse, rollback ownership, settings sweeps and compaction preflight |
| 06 — probability, TOP_K and GDN | Rejection verification, lazy grammar, wave32 selection, FP32 recurrence and rejected target shortlist |
| 07 — power and thermals | Auto versus high, real service behavior, idle cost and rejected controls |
| 08 — serving and clients | Generic server command, reasoning/sampling, tools, projector and application limits |
| 90 — results ledger | Dated comparisons, qualifications and evidence identifiers |
| 99 — sources and verification | Public source references, implementation provenance, neutral studies and correctness gates |
What we kept #
- A checksum-pinned Prism HIP/gfx1100 runtime and the original PQ2 target tensors with a merged, trained r3 Q8 MTP head.
- One 131,072-token slot; Q8 keys/Q4 values and compiled mixed-KV FlashAttention in target and draft; full GPU weight offload.
- PQ2 unpack-once sharing across 2–4 verification columns; fused large-J scale staging, smaller large-J tiles and two output rows with correct expert tails.
- Ngram reuse before depth-two MTP, plus only rolling speculative checkpoints on the GPU under explicit architecture/lifetime guards.
- Probabilistic rejection verification when the request and lazy grammar permit it; full-vocabulary wave32 HIP TOP_K; eligible FP32 chunked GDN prefill.
- Guarded zero-output MTP catch-up that computes only its identical K/V stores, and cache-owned immutable device rotation matrices with exact reset/restore lifetime and accounting. No hidden-state precision cut or sync skipping.
- The official Q8 vision projector without sacrificing the text/context/thinking contract. Image input is optional if your workload is text-only.
- Native auto GPU power policy. The tested high profile was rolled back; it is an optional efficiency tradeoff, not a retained speed optimization.
How fast? #
The 2026-10-14 selected pinned-base implementation adds modest exact catch-up/input-lifetime improvements, not a breakthrough. Independent MTP screens repeatedly improved 33K/99K prefill; a combined long-context screen and final exact-artifact confirmation preserved outputs/counters and reduced request wall. Short native-shaped effects were mixed, so no universal speedup is claimed. The ledger contains exact rates, paired effects, sample counts, earlier regressions and the distinction between component screens and combined confirmation; do not multiply their gains.
Naturally completed native tool/image and 95.8K distributed-fact tasks, additive retained-prefix hits and a six-minute sustained run extend the behavioral evidence. They do not prove arbitrary full-128K quality, maximum output length, hours-long thermals or performance on another GPU. The sustained reference GPU was hot; no fan/clock/power change was retained. Earlier exact-copy rates measure reuse, not novel generation.
The separate October 18 native-context screen completed a distributed archive task through 228K input with full 32K output headroom. No collapse appeared in that bounded task, but generation slowed substantially and cold ingestion became expensive. The reference default stays 128K; this is capacity evidence, not a recommendation to maximize context.
The same October 18 follow-on built four HIP/runtime candidates. Activation reuse repeated small short-prompt wins, but final longer-input results were mixed; the other candidates also had slower samples. None was installed. The selected runtime remains the qualified October 14 profile rather than a collection of favorable microbenchmarks.
A separate draft-efficiency screen tested packed draft-only vocabulary subsets and per-position draft-temperature calibration. Isolated projection work and estimated probability overlap improved, but held-out generation included regressions. Both tested configurations were rejected; target vocabulary, proposal temperature and installed profile stayed unchanged. 06 explains the safeguards and why a favorable microbenchmark or overlap estimate is not a speed qualification.
Completed follow-on lessons — 2026-10-14 #
The initial target-shortlist and compaction preflight did not change the profile. The later compatible-base study separately qualified guarded MTP catch-up and immutable KV rotation mechanisms; it rejected a broad newer branch and the shared-Gram GDN kernel. Actual combined-artifact selection and client evidence are dated in the ledger, not attributed retroactively to the October 11 build. A guarded GPU target shortlist passed bounded equality/oracle checks but produced no meaningful native-shaped request-wall gain and was rejected; shrinking whole-trace transfer bytes did not prove a decode critical-path win. See 06 for the filtered-logits API pitfall and 90 for the same-prototype flag0/1 results.
Two historical consumer compactions showed expensive full-prompt rebuilds, not measured retained-suffix savings. 05 explains why IMROPE, GDN recurrence, draft/hidden state, checkpoints and vision prevent a transparent generic KV splice, and how to measure cache lifetime first. No approximation was deployed. 02 distinguishes merged upstream Hadamard/MTP support from support for the selected private PQ2 codec. A newer compatible Prism candidate was subsequently built, but failed one canonical server contract; no wholesale stock-mainline replacement was qualified. The separate executed native compaction probe measured exact P/S/K and preserved retained entry IDs/facts, while retaining exact re-prefill.
Evidence boundary #
The October9–11 measured profile has FangVoice source provenance
b29cf86956a700dc0697d5af0b4cc5fe0b6f845b, dated optimization records and
retained benchmark summaries. The separate October14 completed studies and
public upstream source observations are identified in 99;
their separate artifacts and gates do not rewrite the earlier qualification.
Historical measurements retain their original dates, workload qualifications and
limits. This Markdown-only recipe documents measured results and transferable
mechanisms; reading it does not establish performance or universal task quality
on a different runtime, client or machine.
No raw user conversations, identity prompt, private service credentials, model files, runtime binaries, copied patches or throwaway benchmark scripts are bundled. The source manifest distinguishes public upstream references from workstation-only evidence.