Researchers debate optimal quantization: larger quantization fits hardware but may slow model reasoning; prompts and token generation speed may not compensate for longer thinking blocks.
Read the original at www.reddit.com→Do anybody has the same feel? Going to lower quants, thinking models tend to think more and more. So much that the speed advantage is cancelled, or it becomes actually slower. The increase in prompt processing speed...
Original headline: "At this point, just use the large quant your hardware fits"
Coverage timeline
- Oct 22, 10:18 UTC r/LocalLLaMA lead source At this point, just use the large quant your hardware fits