New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K
Read the original at www.reddit.com→Looks quite interesting. Lots of numbers presented, but for a single model only. Highlights: File sizes are the same as Q-quants. KLD improves substantially over Q-quants. Performance suffers slightly Creating IQ...
Coverage timeline
- Oct 18, 10:06 UTC r/LocalLLaMA lead source New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K