KLD is misunderstood: it measures how different probability distributions of base and quantized models are across tokens, not a universal quality metric

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't. So what does KLD actually measure? If...