How much difference does KV cache quantization make?
Read the original at www.reddit.com→I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even...
Original headline: "how much difference does kv cache quants even make?"
Coverage timeline
- Oct 25, 12:41 UTC r/LocalLLaMA lead source how much difference does kv cache quants even make?