Massive B70 spec-decoding improvement as Intel XMX matrix engines refactored in llama.cpp Pull Request 29864
Read the original at www.reddit.com→This made a huge improvement to speculate decoding speeds on my B70 running SYCL. Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2. NOTE: Due to the significantly reduced cost of token...
Original headline: "Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp"
Coverage timeline
- Oct 22, 18:16 UTC r/LocalLLaMA lead source Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp