Aggregate scores can hide substantial changes in individual verdicts when LLM judges' instructions vary, study finds.
Read the original at arxiv.org→arXiv:2610.30512v1 Announce Type: new Abstract: LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the...
Original headline: "Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals"
Coverage timeline
- Oct 23, 04:00 UTC arXiv cs.CL lead source Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals