What quantization bits do practitioners actually use day to day, and is 2-bit or 3-bit usable in practice?
Read the original at www.reddit.com→Curious what everyone runs in practice: - What quant do you use most, and for what (chat, coding, agents)? - 2-bit: do you use it? Is it actually usable or too broken? - 3-bit: good enough for coding and more complex...
Original headline: "What quant do you actually use day to day? Anyone happy with 2-bit or 3-bit?"
Coverage timeline
- Oct 18, 13:05 UTC r/LocalLLaMA lead source What quant do you actually use day to day? Anyone happy with 2-bit or 3-bit?