Ternary Bonsai 2 runs with native CUDA offload and fast CPU in ik_llama.cpp, bringing full compatibility outside PrismML’s fork.
Read the original at www.reddit.com→Hey folks, Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2! Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256...
Original headline: "[Guide] Ternary Bonsai 2 is finally 100% working on ik_llama.cpp (Native CUDA offload + Fast CPU -rtr)"
Coverage timeline
- Oct 31, 15:01 UTC r/LocalLLaMA lead source [Guide] Ternary Bonsai 2 is finally 100% working on ik_llama.cpp (Native CUDA offload + Fast CPU -rtr)