Knowledge distillation in small instruction-tuned LMs has asymmetric effects on bias, improving context-following for unambiguous tasks but degrading calibration for ambiguous tasks.
Read the original at arxiv.org→arXiv:2607.28639v1 Announce Type: new Abstract: We show that knowledge distillation in small instruction-tuned language models has asymmetric effects on bias. On unambiguous tasks (BBQ-disambig), response-based...
Original headline: "The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models"
Coverage timeline
- Aug 3, 04:00 UTC arXiv cs.CL lead source The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models