arXiv cs.CLPaper
Your Benchmark Is an Empirical Measure Over Difficulty
This is important meta-science. Many benchmarks weight certain difficulty ranges heavily while leaving gaps elsewhere, so leaderboard rankings can mislead. For researchers and builders, this means: don't trust aggregate benchmark scores to tell you where your model is weak. Profile difficulty distribution yourself. The paper's diagnostic metrics are useful for anyone designing new benchmarks.