r/LocalLLaMA ·1d ago

GPQA, MMLU-Pro, and MMMU-Pro audits find up to 12% of questions broken; cleaned versions released

I was very curious why all the models were topping out on GPQA-Diamond around 92 or 93% (AA) and spent the last few weeks pouring over GPQA (Diamond and Extended), and then expanded to auditing MMLU-Pro and MMMU-Pro....

Original headline: "[PAPER] GPQA, MMLU-Pro, and MMMU-Pro were audited for broken questions, and up to 12% of them had to be removed. New drop in clean versions released"

Coverage timeline

Jul 28, 19:58 UTC r/LocalLLaMA lead source [PAPER] GPQA, MMLU-Pro, and MMMU-Pro were audited for broken questions, and up to 12% of them had to be removed. New drop in clean versions released