CUDA shows big speedup for low-bit MMVQ types like Q2_K and Q3_K in llama.cpp pull request 30517
Read the original at www.reddit.com→big speedup, but only for specific quantizations like Q2_K and Q3_K submitted by /u/jacek2023 [link] [comments]
Original headline: "CUDA: faster MMVQ for low-bit types by am17an · Pull Request #30517 · ggml-org/llama.cpp"
Coverage timeline
- Oct 17, 14:33 UTC r/LocalLLaMA lead source CUDA: faster MMVQ for low-bit types by am17an · Pull Request #30517 · ggml-org/llama.cpp