Qwen 27B sacrifices context to run better quants; pros and cons
Read the original at www.reddit.com→I was running Unsloth IQ4_XS for a while and while it provided me with a healthy 96K context on 20GB vram, I did notice all the issues it had during draft and all the time and local tokens wasted trying to fix said...
Original headline: "Qwen 27B - sacrificing context in order run better quants, pros and cons."
Coverage timeline
- Oct 16, 17:15 UTC r/LocalLLaMA lead source Qwen 27B - sacrificing context in order run better quants, pros and cons.