Benchmarks misrepresent model difficulty by overweighting certain regions and leaving parts of the difficulty axis uncovered

Very interesting paper on the way that benchmark question difficulty and framing affect the observed improvement in scores for model training. “Our results show that many benchmarks implicitly overweight certain...