How much can we trust leaderboard rankings? A new paper on arXiv, "The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models", provides the most rigorous answer to date: not very much. The authors construct a formal geometric theory of what benchmark suites can and cannot distinguish about model capabilities, and the results are sobering for anyone who has cited a leaderboard position as evidence of model superiority.
The core problem is one that practitioners have suspected informally for years. Benchmark suites are finite, and the capability space of a language model is not. The paper turns this vague concern into a precise mathematical statement, bounding exactly how different two models can be while appearing identical on any given suite. The bound is not small.
Key Contributions
The paper makes three distinct contributions that are worth separating clearly.
- An indistinguishability bound for benchmark suites. For any suite with effective dimensionality deff, two convex capability profiles consistent with the same scores can differ in Hausdorff distance by up to epsilon + CR m-1/(d_eff - 1). A matching Lipschitz lower bound confirms this rate is tight. The effective dimensionality is estimated empirically from the score matrix via participation ratio of the eigenvalue spectrum.
- A submodular greedy algorithm for benchmark selection. Using the Nemhauser (1 - 1/e) guarantee, the authors identify a stable core of benchmarks that maximises spectral coverage. The trained subset transfers across temporal quarters with 93-97% retention, which is a non-trivial stability property.
- Resolution of Gardner's Problem 1.5 for C² bodies. This is presented as an independent mathematical contribution. For convex bodies with C² support functions and Gauss curvature bounded below by kappa, the minimax recovery rate from m width measurements in general dimension D is Theta(R / (kappa m2/(D-1))). The planar case gives Theta(m-2), which resolves the problem as Gardner originally stated it.
The connection between the third contribution and the first two is not merely analogical. Benchmarks are modelled as width measurements of the convex hull of a model population in capability space. The geometric tomography machinery that resolves Gardner's problem is exactly the machinery that produces the coverage bounds. The mathematical unification is clean.
Methodology and Empirical Validation
The width model treats each benchmark as a scalar, Lipschitz, conservative measurement of the population's capability profile. The linearisation hypothesis is validated empirically: across all 12 benchmarks in the extended suite, a linear regression on the top 5 principal components achieves R² of at least 0.795 (median 0.946), with the gap to a quadratic model never exceeding 0.067. MMLU specifically achieves R² = 0.898 under this model, with score-difference predictions correlating at r = 0.95 with actual differences over random model pairs. The width model is not just a convenient fiction.
Three independent leaderboards are analysed: Open LLM v2, an extended 12-benchmark suite, and LiveBench. All three have deff between 2.86 and 4.80 on their competitive frontier. This is strikingly low. The structural blind spot implied by this dimensionality exceeds the observed score gap between the top two models by two orders of magnitude and dominates statistical noise by 52 to 127 times.
The ranking instability results are particularly striking. Under a chi-squared projection model, a 500-trial random visible/held-out split shows that 92% of trials swap the top-1 ranking, with an average of 2.83 of the top-5 models changing position. The half-split swap rate stays between 0.38 and 0.49 across six hidden-capability priors and four ambient dimensions. The isotropic prior turns out to be the optimistic case, via a Schur-convexity argument. Real capability distributions are likely more concentrated, making the actual situation worse than these estimates suggest.
The ambient dimension D is honestly treated as unidentified. Three estimation methods (eigenvalue power-law extrapolation, cross-validation against swap rates, parallel analysis) give a range of D from 6 to 184, and the authors do not pretend otherwise. The qualitative conclusion, that the top model is not separable from its runner-up, holds across this entire range.
Results and the Evaluation Monoculture Problem
The greedy coverage algorithm identifies a stable core of 4 benchmarks (MUSR, GSM8K, IFEval, MMLU on the extended suite), with 7 of 12 sufficing for 90% coverage. The eigenstructure predicts which evaluations are irreplaceable (rho = -0.69, p = 0.013 for removal disruption) and which external evaluations bring genuinely new information (rho = +0.38 for Arena categories).
A finding that deserves attention beyond the benchmark-selection application: three fundamentally different evaluation methodologies (accuracy benchmarks, human pairwise preference, and LLM-as-judge) produce highly correlated rankings, with mean Spearman rho = 0.75. The g-factor dominates across evaluation types, not just within benchmark suites. This means the blind spot is not an artefact of using one type of evaluation. Capability dimensions orthogonal to the shared ranking axis are unmeasured by all methodologies simultaneously.
The paper also draws a useful distinction between width measurements (aggregate benchmark scores) and X-ray measurements (item-level results). Theorem 20 shows that non-adaptively, X-rays do not improve the polynomial stability rate over widths. The angular gap between measurement directions is the universal bottleneck. Adaptive evaluation is the only path to a substantially better rate, which connects directly to the IRT approach of Polo et al. (2024) and suggests a natural two-level optimisation: the greedy algorithm selects which benchmarks to include at the outer level, while IRT selects which items to run within each benchmark at the inner level.
Limitations and Implications
The authors are careful about what the theory does and does not claim. The bounds concern what benchmarks can distinguish, not what models actually do in deployment. The convexity assumption is conservative (non-convex profiles enlarge the bound). The coverage bridge requires approximate isotropy. Shared items between benchmarks deflate deff and understate the blind spot; controlling for log(params) raises deff from 4.80 to 5.11, confirming the main-text estimates are conservative.
There is also a real risk of misreading the coverage results. The finding that 7 of 12 benchmarks suffice for 90% coverage is not a licence to evaluate on fewer benchmarks. The 10% blind spot remaining is precisely where Theorem 2 shows meaningful capability differences can hide. The optimal subset is population-dependent and must be recomputed as the model landscape evolves. Redundancy in benchmarks provides robustness to overfitting that is orthogonal to coverage.
For the Gardner's problem contribution, the authors are appropriately careful. The full problem as stated by Gardner concerns X-ray stability rather than width stability, and the paper notes that the X-ray case for arbitrary dimensions remains open. The planar case is resolved, and the general-dimension width result is new and tight, but the claim of full resolution should be read with the scope limitations in mind.
The practical recommendations are concrete: treat deff as a leaderboard-design diagnostic, use the chi-squared bound as a rank-claim rejection threshold, run the greedy coverage algorithm before adding new benchmarks, and keep the stable core permanent while rotating the remainder. These are actionable without requiring adoption of the full theoretical apparatus.
The paper is technically dense but the empirical validation is thorough, with robustness checks across populations from n = 49 to n = 4,576 and across three non-overlapping suites sharing zero benchmarks. Per-theorem falsification criteria are provided in the appendix, which is an unusually welcome commitment to testability. Read the full paper at arXiv:2606.05169.