llama-cpp-bonsai
llama.cpp fork for Ternary Bonsai 2 27B: MTP speculation, suffix cache reuse (SCR), q8 KV, two-slot fairness, fused DeltaNet kernels
Star
Fork
Watch

Commits on main
cache: conversations are stored against their learned prefix
committed
cache: learning outlasts restarts, pins after two prompts, learned prefixes stay in RAM
committed
cache: keep learned prefixes learned through the entry api
committed
cache: learned prefixes are restore points for splices
committed
cache: learn the prefix that distinct prompts share
committed
cache: replay from a learned prefix inside reused turns
committed
cache: learn and pin repeated system prefixes
committed
scr: retain up to 64 matching spans for fragmented transcripts
committed
scr: flush pending MTP rows before reused runs and aborts
committed
docs: add README and standalone patches archive
committed
bonsai: relax prefix cache and SCR floors for one-shot services (cache-tuning)
committed
bonsai: drop rejected MTP rows on fair step to prevent draft desync (mtp-fair-deferred)
committed
bonsai: q8_0 decode/verify flash attention raw K/V lead staging (fattn-verify-lead)
committed
bonsai: fused CUDA kernel for DeltaNet conv path (gdn-conv)
committed
bonsai: in-place GDN recurrent state reading from cache rows (gdn-rows)
committed
bonsai: two-slot owed-time fairness scheduler (fairness)
committed
bonsai: PTQ1_0 tensor-core verify mat-vec register reduction (decode-verify)
committed
bonsai: per-request telemetry and latency metrics (telemetry)
committed
bonsai: image embedding cache and NVMe .kvc v3 format (imgcache)
committed
bonsai: suffix cache reuse (SCR) for linear attention hybrids (scr)
committed
bonsai: prompt family classification by system prompt block (cache-family)
committed
bonsai: prompt cache retention policy, classes, and learned families (cache-policy)
committed
bonsai: HTTP cache inspection and eviction API (cache-api)
committed
bonsai: pair-tiled 6-head GQA q8_0 prefill flash attention (fattn-q8-prefill)
committed
bonsai: PTQ1_0 MMQ main loop staged per 128-value block (mmq-staged)
committed
bonsai: Hadamard transform writes per-128 q8 tiles directly (fwht-q8)
committed
bonsai: PTQ1_0 MMQ 128-value activation scaling and int32 accumulation (mmq-y128)
committed
bonsai: PTQ1_0 MMQ cp.async double-buffering (mmq-pipeline)
committed
bonsai: prefill kernel fusions (ADD+RMS_NORM+MUL and SWIGLU+FWHT) (prefill-fusions)
committed
bonsai: NVMe disk tier for prompt cache with DeltaNet checkpoints (prompt-cache-disk)
committed
bonsai: publish h for output rows to unblock MTP draft chain (mtp-chain-h)
committed
bonsai: PTQ1_0 tensor-core mat-vec for 3-8 columns (ptq1-tc-verify)
committed
bonsai: in-place q4_0/q8_0 flash attention on tensor cores (q8-kv)
committed
bonsai: MTP catch-up rows ride in next draft batch (mtp-decode)
committed
bonsai: GDN prefill warp reductions with ahead next-token loads (gdn-prefill)
committed
bonsai: optimize time-to-first-token on cached prompts (ttft)
committed
bonsai: batch-invariant mode over F16 cache (bi-mma-f16)
committed
sudoingX
Test: add head-size-256 decode cases for the FA vector kernel
committed
sudoingX
Fix: keep the q4_0 V dequant of the FA vector kernel in registers
committed
sudoingX
Test: add a PTQ1_0 mat-vec case that fails when the kernel skips its PDL wait
committed
sudoingX
Fix: call ggml_cuda_pdl_sync before the PTQ1_0 mat-vec reads its activations
committed
sudoingX
Fix: cap the PTQ1_0 mat-vec at 4 columns and send 5 and above to the MMQ tile path
committed
sudoingX
Docs: scope the GGML_CUDA_BATCH_INVARIANT comment to the paths the flag changes
committed
sudoingX
Fix: budget the PTQ1_0 mat-vec shared memory the launch will request
committed
sudoingX
Docs: state the batch-invariance guarantee as 1 to 4 columns
committed
sudoingX
Fix: keep GGML_CUDA_RESTRICT off the PTQ1_0 mat-vec kernel signature
committed
sudoingX
Fix: use the mat-vec kernel for bf16 matrices under 64 rows at 2 to 8 columns
committed
sudoingX
Add: GGML_CUDA_BATCH_INVARIANT for batch-invariant small-batch kernels
committed
sudoingX
Test: add Bonsai 2 projection shapes to the mul_mat perf cases
committed
sudoingX
Add: dedicated PTQ1_0 mat-vec kernel with full lane utilization
committed
sudoingX
Add: planar-transposed activation layout for the PTQ1_0 mat-vec path
committed
Scott Guyton
x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 (#248)
committed
bri-prism
tests : build test-dspark-forward with MSVC (#273)
committed
Chirag Patnaik
spec : add DFlash2 support (local convolution + candidate selector) (#27342) (#27816) (#261)
committed
bri-prism
server: make speculative decoding visible when it is enabled but not drafting (#203) (#272)
committed
Jason Titus
metal : add PTQ1_0 mat-vec for two to four columns (#262)
committed
lenny76
ggml-cpu : AVX-512 tiled kernel for single-token flash attention (#254)
committed
lenny76
ggml-cpu : PQ2_0 AVX-512 VNNI gemm on tiles of 8 activation rows (#256)
committed
bri-prism
release: ship CPU backend variants in the CUDA and HIP bundles (#241) (#271)
committed
Pasha Khosravi
ggml-cpu : fix PQ2_0 x86 vec_dot build on non-AVX2 CPU variants (#263)
committed