48 DeltaNet linear-attention layers + 16 full-attention layers (with interleaved multi-axis RoPE)
Ternary weights quantized via PTQ1_0 with Hadamard rotation
MTP (Multi-Token Prediction) draft head (n-max 2)
Vision encoder: mmproj-Q8_0 (Qwen2.5-VL architecture)
Context window: 90,112 tokens across 2 concurrent slots (-c 90112 -np 2 -kvu)

The Production Patches #

The patch series is applied in sequence on main and archived standalone in patches/:

Speculative Decoding & MTP #

mtp-decode: MTP catch-up rows ride in the next draft batch; conv-state concat uses transposing kernel (+6% decode).
mtp-chain-h: Publish hidden states h for output rows to unblock second-step MTP conditioning (acceptance 56% → 73%).
decode-verify: PTQ1_0 tensor-core verify mat-vec folding residue slots in registers with warp-reduce sums (+8.7% decode).
ptq1-tc-verify: Tensor-core verify mat-vec for 3 to 8 columns via mma.m16n8k32.
mtp-fair-deferred: Drops rejected MTP rows on fair step to prevent draft context desynchronization under fairness.

CUDA Kernels & Matmul Accelerations #

gdn-rows: DeltaNet reads recurrent state in place from cache rows directly via src[6], eliminating state gather (+2.3% decode).
gdn-conv: Fused CUDA kernel for concat -> ssm_conv -> silu -> conv-state writes (+1.5% decode).
gdn-prefill: DeltaNet prefill column-batched warp reductions with ahead next-token loads (~1.9x faster per ubatch).
bi-mma-f16: Batch-invariant tensor-core evaluation for 1–4 query tokens over F16 cache.
q8-kv: In-place q8_0 / q4_0 flash attention on tensor cores under batch-invariant mode.
fattn-q8-prefill: Pair-tiled 6-head GQA q8_0 prefill flash attention (+8% at 16K, +14% at 32K, +21% at 64K).
fattn-verify-lead: Flash attention q8_0 decode/verify tile stages raw K/V with cp.async and starts next tile a phase early (+2.9% at 64K).
mmq-pipeline: PTQ1_0 MMQ activation double-buffering with cp.async (+3.3% matmul).
mmq-y128: PTQ1_0 MMQ activation scaling per 128 values and int32 accumulation (+11% prefill).
fwht-q8: Hadamard transform writes per-128 q8 tiles directly (+3% prefill).
mmq-staged: PTQ1_0 MMQ staged loop per 128-value block (+10% prefill).
prefill-fusions: Bit-identical fusions (ADD+RMS_NORM+MUL and SWIGLU+FWHT).

Cache System, Suffix Cache Reuse (SCR), & Multimodal #

prompt-cache-disk: NVMe disk tier (.kvc) saving RAM evictions and DeltaNet checkpoints (8K restore in ~0.6 s vs ~14 s prefill).
imgcache: Image embedding cache (256 MiB RAM + 4 GiB disk) and .kvc v3 chunk map.
cache-api: HTTP API for cache inspection (GET /cache) and targeted eviction (POST /cache/clear).
cache-policy: Per-entry classes (skip, ephemeral, default, keep, pin), auto-classification, and TTL eviction.
cache-family: Learned prompt families hashed from system blocks (one-shot services learn ephemeral after 8 calls).
scr: Suffix cache reuse for mid-prompt edits (prompt compaction, system edits) with approximate linear-state splice.
cache-tuning: Lowered SCR floor to 128 tokens; retains warm system prompt prefix in VRAM across one-shot API calls.
ttft: Eliminates scheduler reserves and redundant checkpoint rewrites on cached prompts.

Scheduling & Telemetry #

fairness: Two-slot owed-time fairness scheduler: interleaves 1024-token prompt chunks with single-step decodes (~16–22 t/s interactive decode during 30K prefills).
telemetry: Low-overhead per-request telemetry ring buffer and JSONL logging (/stats, /stats/requests).
scr-mtp-flush: Flush MTP catch-up rows before each SCR run and abort, before disabling the splice guard. This prevents position errors after 1–3 token edited gaps.
scr-spans: Allow up to 64 matching spans for fragmented transcript inputs.
prefix-learn: Learns repeated system prompts. After three chat requests repeat the exact tokens before the first user message, that prefix state is pinned as its own cache entry (at most 8, least recently used dropped first), apart from the conversation history. A longer cached conversation still wins. Regression test: tools/server/tests/prefix_cache_test.py.
prefix-replay: When cached text already runs past a system prefix due to be learned (an agent resending its early turns), the request reprocesses from that prefix once, if at most 8192 tokens follow it, so the prefix state can be stored.
prefix-shared: The learned prefix is the longest start that three distinct prompts share (told apart by their first user message), not only a whole system block, so per-session text late in a system block (an agent's name, a memory snapshot) still leaves the shared start pinned.
prefix-restore: A learned prefix also keeps its recurrent state as a checkpoint, and SCR plans may restore from it. A splice that re-decodes a shared start (agents whose system blocks differ at their name) stores that checkpoint, so the next one resumes there instead of re-prefilling from token 0.
prefix-entry: Setting a class on a learned prefix (POST /cache/entry, the status page's pin and keep buttons) or a family verdict keeps it learned; before, a checkpoint-only prefix became an ordinary entry without a state and a later request restored nothing from it. Restoring an entry without a state is refused, and an entry is consumed only after both target and draft states are restored.
prefix-persist: The prompts seen toward learning are kept in prompts-seen.bin beside families.json, so a restart does not start learning over; two distinct prompts sharing a prefix are enough to pin it (at most 16 pins). Learned prefixes are written to disk, read back into RAM at startup and never evicted from RAM, so a checkpoint-only restore point also outlasts a restart.
prefix-share: A conversation that continues a learned prefix is stored against it, in RAM and on disk: its state keeps only the KV rows whose bytes differ from the prefix's and its own recurrent state, and is rebuilt byte for byte when restored (server-kv-share.cpp). Deleting a prefix rebuilds the conversations in RAM that use it and removes those on disk. GET /cache names an entry's prefix in shared_from.

SCR reuses exact token spans, not semantic similarity. The current planner moves at most 64 spans, each at least 256 tokens; its final span must end at an available recurrent-state snapshot. The 128-token threshold in cache-tuning is the total-plan acceptance floor, not the minimum span length. A summary request that reformats a transcript may therefore reuse only part of its input.

Production Deployment (Valefar) #

Runtime Flags #

llama-server   -m /path/to/Ternary-Bonsai-2-27B-PTQ1_0-mtp-valefar.gguf   --mmproj /path/to/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf   --image-max-tokens 1024   --alias bonsai-2-27b   -c 90112   -ngl 99   -fa on   -np 2   -kvu   -ctk q8_0   -ctv q8_0   -ctkd q8_0   -ctvd q8_0   -ub 1024   -bs   --jinja   --reasoning-effort medium   --temp 1.0   --top-p 0.95   --top-k 20   --spec-type draft-mtp   --spec-draft-n-max 2   --scr   --cache-ram 24576   --cache-disk /var/cache/bonsai2   --cache-disk-size 51200   --host 127.0.0.1   --port 8899

Verified Performance (RTX 3060 12GB) #

Decode Speed: ~60.3 t/s pooled across real workloads (code, prose, agent, summary); ~71 t/s single slot.
Draft Acceptance: ~70–81% acceptance with MTP n-max 2.
Prefill Speed: ~650 t/s short, ~710 t/s at 16K, ~445 t/s concurrent with decoding.
VRAM Usage: Peak 11,793 MiB with multimodal vision encoder + 17K prompt context (169 MiB free VRAM headroom).
Two-Slot Fairness: Maintains 15.9 t/s interactive decode on slot 0 while slot 1 evaluates a 30,000-token prefill.

Building #

Standalone (CMake + Ninja) #

cmake -B build -G Ninja   -DGGML_CUDA=ON   -DCMAKE_BUILD_TYPE=Release   -DCMAKE_CUDA_ARCHITECTURES="86;89"   -DLLAMA_CURL=OFF   -DLLAMA_BUILD_UI=OFF

cmake --build build --target llama-server -j $(nproc)

Nix / NixOS #

The standalone derivation is included in patches/llama-cpp-bonsai.nix.

Clone this repository

llama.cpp (Bonsai Edition) #

Architecture & Features #