Evaluating AI systems on mathematical reasoning has become significantly harder as benchmarks shift from short numerical answers toward full natural-language proofs. Scoring those proofs reliably requires either formal verification (still limited in coverage) or a human-equivalent judge, and frontier LLM judges are expensive enough to constitute a real research tax. This paper, available at arXiv:2608.00004, asks a pointed practical question: can cheap open-weight models substitute for frontier judges when grading proofs against a provided reference solution and rubric? The answer, on the IMO-GradingBench dataset, is largely yes.
What the Paper Contributes
The core contribution is an empirical comparison of three cheap judges (GPT-OSS-120B, DeepSeek-V4-Flash, Gemma-4-31B) against two frontier baselines (Claude Opus 4.7, Gemini 3.1 Pro) on pass/fail grading of Olympiad proofs. The benchmark, IMO-GradingBench, consists of 1,000 instances each containing a problem, a reference solution, a candidate proof, and an expert human score on the standard 0-7 IMO scale. The primary metric is pass-agreement: whether the judge's binary pass/fail decision (threshold at 6/7) matches the human grader's.
Secondary contributions include:
- A test of majority-vote consensus across the cheap trio, motivated by prior work showing panels of small models can outperform single large judges
- A post-hoc rule search on the full 1,000-instance benchmark exploring unanimous agreement (all-three-pass) as an alternative consensus rule
- Run-to-run variance analysis across four replicate runs, revealing stability differences between individual and ensemble configurations
Methodology and Experimental Design
Each judge receives the problem statement, the ground-truth reference proof, and the candidate proof, then predicts a score. The setup is deliberately reference-based: this is not reference-free verification, and the authors are careful to frame it as a judging study rather than a solving study. The distinction matters because, as they note and as concurrent work by Naik et al. supports, the reference solution appears to do significant epistemic work for the judge. In the reference-free setting, cheap judges trail the frontier by around 10%; that gap closes in the reference-based setting studied here.
The validation comparison (n=200) is the only setting where all five judges run head-to-head, due to frontier API costs. The full benchmark (n=1,000) is used only for the cheap-tier extended analysis. Confidence intervals are 95% bootstrap intervals over 1,000 resamples, though the authors acknowledge these do not account for clustering by source problem (30 problems in the pool), meaning the reported intervals are somewhat optimistic.
Main Results
On the 200-instance validation set, the pass-agreement figures are:
- GPT-OSS-120B: 0.875 [0.830, 0.920], cost $0.32 per 200 calls
- DeepSeek-V4-Flash: 0.860 [0.815, 0.910], cost $0.70
- Cheap consensus (majority vote): 0.855 [0.810, 0.905], cost $1.73
- Claude Opus 4.7: 0.855 [0.805, 0.900], cost $32.45
- Gemini 3.1 Pro: 0.840 [0.790, 0.890], cost $28.61
- Gemma-4-31B: 0.795 [0.740, 0.850], cost $0.71
Every cheap judge's confidence interval overlaps the frontier leaders' point estimates. The cost differential is roughly 100x for the top cheap judge versus the frontier baselines. The majority-vote consensus performed well but did not beat GPT-OSS-120B individually, which runs counter to the prior expectation from Verga et al. (2024) that panel consensus would be the safest budget option.
The more interesting finding comes from the full-benchmark rule search. Requiring unanimous agreement across all three cheap judges (all-three-pass) achieves the highest precision and the smallest run-to-run variance across four replicate runs. This makes intuitive sense: unanimity is a conservative rule that filters out borderline cases where individual judges disagree, trading recall for precision and stability. The authors recommend this as a deployable default while being transparent that it was identified post-hoc.
Limitations and Open Questions
The paper is commendably honest about its limitations. Several are worth highlighting for readers considering adopting these findings.
Post-hoc rule selection. All-three-pass was identified through rule search on the same data it was evaluated on. The precision and stability advantages are real within this dataset, but the recommendation needs independent replication before it should be treated as a general default. The authors say this explicitly, which is the right call.
Sample size and clustering. 200 instances for the head-to-head comparison is enough to establish rough parity but not enough to rank the models confidently. The bootstrap intervals do not account for problem-level clustering (30 source problems), so they are narrower than they should be. A cluster bootstrap would widen them.
Run-to-run variance. Individual cheap judges vary by roughly 2 percentage points across runs due to provider routing variability through OpenRouter. The frontier baselines were not replicated comparably, so the stability comparison is asymmetric.
Scope. These results are specific to reference-based grading of IMO-style competition mathematics. The authors explicitly do not claim transfer to reference-free verification or other domains, which is the correct epistemic position. Whether the finding holds for research-level mathematics (where reference proofs are less clean and rubrics less standardised) is genuinely open.
Missing frontier models. GPT-5.5 Pro at high reasoning settings was out of budget. Given that the cheap tier already matches Opus and Gemini Pro, it is possible (the authors acknowledge this) that a sufficiently powerful frontier model would pull ahead. The comparison is against a specific slice of the frontier, not the frontier ceiling.
For researchers building evaluation pipelines around AI mathematical reasoning, the practical upshot is clear: if you have a reference solution and a human rubric, a cheap open-weight judge is a credible substitute for an expensive frontier call, and running three cheap judges under a unanimity rule gives you better precision and more stable results across runs. The cost saving is large enough to matter for any system that iterates judgment in a loop, such as self-improving proof generation systems where a judge call occurs at every revision step.
The finding that majority vote did not outperform the best individual member is worth registering against the Panel-of-LLM-evaluators literature. That prior result was demonstrated on QA and chatbot tasks; it may not transfer cleanly to structured grading tasks where a provided reference already constrains the judgment space. The unanimous rule recovering stability without recovering a precision advantage from simple majority vote suggests the benefit of ensembling here is primarily about variance reduction rather than bias cancellation.
Full details, tables, and the forest plot comparison are in the paper: arXiv:2608.00004.