The standard response to a saturated AI benchmark is straightforward: retire it and build a harder one. ARC-AGI became ARC-AGI 2 and 3; MMLU became MMLU-Pro; HumanEval became HumanEval+. This retire-and-replace cycle has become so routine that it is rarely questioned. A new paper, Life After Benchmark Saturation: A Case Study of CORE-Bench, argues that this convention is not just inefficient but actively harmful to our understanding of agent capability. The authors make a compelling case that accuracy saturation and benchmark saturation are not the same thing, and conflating the two discards a great deal of signal.
The argument matters because benchmark accuracy has become the dominant currency of AI evaluation. When top agents cluster near ceiling scores and become statistically indistinguishable, the field interprets this as the benchmark being exhausted. But the authors point out that accuracy is only one of several dimensions along which agents differ, and often not the most practically relevant one. The paper uses CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate what remains measurable after accuracy plateaus.
Key Contributions
The paper makes three substantive contributions, each addressing a different failure mode of accuracy-centric evaluation.
First, the authors conduct a systematic audit of CORE-Bench Hard using agent trajectory logs, uncovering 15 task-level errors (incorrect ground truths, malformed questions, grading bugs, and unsolvable tasks) and 20 tasks containing exploitable shortcuts. These issues were difficult to surface before capable agents began solving the benchmark at high rates, since less capable agents fail for so many reasons that shortcut exploitation is hard to isolate. The corrected benchmark, CORE-Bench v1.1, contains 39 tasks. The authors also introduce CORE-Bench OOD, a 19-task suite covering physics, engineering, economics, and computer science, disciplines absent from the original benchmark, to test whether saturated accuracy generalises under distribution shift.
Second, the paper reports multidimensional evaluation results across 20 agent runs using four scaffolds (Claude Code, Codex CLI, OpenCode, and CORE-Agent) and three underlying models (Opus 4.5, Opus 4.6, and GPT-5.4). Even where top-line accuracies are statistically indistinguishable, agents differ substantially in cost, latency, reliability, and failure mode distribution.
Third, the authors run a small randomised experiment measuring the uplift from human-agent collaboration on real-world reproducibility tasks, covering 20 papers across machine learning and the social sciences, with five participant evaluators.
Methodology and Findings
The model-scaffold decoupling analysis is the most technically interesting part of the paper. A standard leaderboard entry collapses model and scaffold into a single accuracy figure, making it impossible to attribute performance gains to either component. The authors hold models constant while varying scaffolds and vice versa, then classify all 56 accuracy failures by root cause using an LLM-assisted rubric.
Several findings stand out:
- Opus 4.5 achieves 82.1% accuracy on both CORE-Agent and OpenCode, yet the two scaffolds disagree on outcomes for 31% of individual tasks. An oracle that always picks the better scaffold for each task achieves 100% accuracy for both Opus 4.5 and GPT-5.4, meaning every task is solvable by at least one scaffold.
- Scaffolds induce fundamentally different solution strategies. With Opus 4.6, Claude Code derives 41% of answers from unmodified code output and only 3% from vision-reading a rendered figure. The same model on CORE-Agent uses vision-reading 31% of the time. For Opus 4.5, the vision-read rate jumps from 3% on Claude Code to 62% on CORE-Agent. Vision-reads used as a fallback after code failure pass only around 50% of the time, compared to 93% when used on a clean run, which explains much of CORE-Agent's accuracy gap.
- Direct targeted fixes succeed 95.2% of the time across 269 instances; rewrites from scratch succeed only 67.8% of the time across 59 instances. Scaffold tendency toward direct fixes closely tracks overall accuracy: Codex CLI uses direct fixes 82% of the time, CORE-Agent only 49%.
The failure taxonomy in Table 5 of the paper is particularly instructive. Wrong-metric errors concentrate in CORE-Agent; timeout and dependency failures concentrate in OpenCode. These are not random noise around a shared mean. They reflect systematic architectural differences in how scaffolds approach problem-solving, differences that accuracy alone cannot capture.
The human-agent uplift study finds that manual reproduction sessions lasted approximately 2.11 times as long as human-agent collaborative sessions (clustered standard error 0.09, p = 0.00176). Five of 25 manual runs hit the three-hour time limit without completing; none of the 25 human-agent runs did. The authors appropriately flag that the estimated speedup is likely conservative for this reason. Agents completed 19 of 25 collaborative runs fully autonomously. The most valued agent contributions were environment setup, running code, identifying main scripts, and navigating documentation quickly.
Limitations and Open Questions
The authors are candid about the limitations of the uplift study. Twenty papers and five participants is a small sample, and all five participants are also coauthors, creating potential demand effects. The lack of verified ground truth for reproduction outcomes means the study measures process-level uplift (time, blocker recovery) rather than outcome correctness. The machine learning papers were drawn from award-winning conference papers, which are likely better documented than average. These are real constraints on how far the uplift findings can be generalised.
There are also methodological questions worth raising about the broader argument. The six dimensions the authors propose (construct validity, out-of-distribution generalisation, efficiency, reliability, model-scaffold decomposition, and human uplift) are not equally tractable to measure across arbitrary benchmarks. CORE-Bench is well-suited to this analysis because it has a direct human counterpart task, clear distribution shift axes, and naturally varying efficiency profiles. Many benchmarks lack these properties, and it is not obvious how to operationalise all six dimensions for, say, a mathematical reasoning benchmark.
The finding that scaffold version alone can shift accuracy by 40% (Codex CLI v0.122 versus v0.130.0 on GPT-5.1, reported in the appendix) is quietly alarming. It suggests that leaderboard entries are often less reproducible than they appear, and that version pinning is a more serious methodological requirement than the field currently treats it.
Implications
The core intellectual contribution here is the distinction between accuracy saturation and benchmark saturation. This is a useful conceptual clarification that the field needs. The retire-and-replace cycle has an obvious appeal: a harder benchmark is easy to justify, easy to publish, and easy to communicate. But it systematically defers the harder work of understanding why agents succeed or fail, which is what practitioners deploying these systems actually need to know.
The model-scaffold decoupling results have direct practical implications. If an oracle router over scaffolds achieves 100% accuracy on tasks where individual scaffolds plateau at around 80%, then scaffold engineering is not a second-order concern relative to model capability. It is at least as important, and possibly more tractable in the near term. The finding that direct fixes outperform rewrites by a large margin suggests a concrete design principle for scaffold developers.
The paper does not fully resolve how the field should institutionalise multidimensional evaluation. Leaderboards are popular precisely because they reduce complexity to a single number. Getting the community to report and compare across six dimensions is a coordination problem as much as a technical one. Still, the case study is convincing enough that benchmark maintainers should treat trajectory-level auditing and failure mode taxonomy as standard practice, not optional extras.
The full paper, code, data, and logs are available at arxiv.org/abs/2606.26158.