As language models take on increasingly complex evaluation tasks, a quiet assumption has crept into standard practice: that giving a capable model more compute, or a larger context window, will make it a better judge. This paper challenges that assumption directly. The core finding is that when a single LLM call must return many independent verdicts simultaneously, some of those verdicts become poorly grounded in the evidence, regardless of how much token or tool budget the model receives. The intervention the authors propose, called sharding, is conceptually simple but has significant implications for how AI oversight pipelines should be designed.
The paper is available on arXiv: Sharding Prevents LLM Oversight Failures and Adversarial Exploitation.
The Problem: Decision Load Degrades Oversight Quality
The authors frame the issue around what they call decision load: the number of independent verdicts a single model call must produce. Their key insight is that this is a distinct variable from computational budget. You can give a judge model more tokens or more tool calls, and it will still be tracking many criteria simultaneously, which appears to dilute the attention each criterion receives.
To test this, they evaluate LLM judges across three expert-labeled datasets spanning genuinely different domains: research replication grading (PaperBench), legal work assessment (JudgmentBench), and clinical trial risk-of-bias evaluation (ROBoto2). Crucially, they compare a holistic single-call judge not just against a sharded panel at the same total budget, but also against a full-budget solo condition where one call gets the entire combined budget of the panel. This control is methodologically important because it isolates the effect of work division from the effect of total compute.
The result is consistent across domains: agreement with expert labels falls as the number of verdicts per call grows, and giving that single call more budget does not recover the lost accuracy. Sharding does.
Methodology: Partitioning Criteria Across Calls
The sharding procedure is straightforward. Given an artifact with K criteria and a base per-call budget B, the criteria are partitioned into S disjoint groups. Each group is assigned to a separate call at budget B, and the resulting verdicts are aggregated. The total budget spent is S times B, identical to the full-budget solo and pooled-opinions conditions.
The authors define four experimental arms with careful budget accounting:
- Solo (holistic): one call, all K decisions, budget B. Cheapest baseline.
- Full-budget solo: one call, all K decisions, budget S times B. Controls for total compute.
- Pooled opinions: S calls each seeing all K decisions at budget B, aggregated. Same total budget as sharded but work is duplicated, not divided.
- Sharded: S calls each seeing K/S decisions at budget B. Same total and per-decision budget as full-budget solo, but lower decision load per call.
The contrast between sharded and full-budget solo is the cleanest comparison: total budget is fixed, per-decision budget is fixed, and only decision load differs. The accuracy advantage for sharding in this comparison cannot be attributed to compute differences.
Adversarial Exploitation and the Presentation Attack
The second major contribution is showing that holistic judge errors are not just random noise; they are systematically exploitable. The authors instantiate a best-of-N presentation attack in which the underlying work is held fixed and only its presentation varies across N submissions. An adversary selects the presentation that maximises the judge's acceptance of genuinely unmet criteria.
The results are striking. Against an overloaded holistic judge, this attack drives over-acceptance severalfold. In a code verification setting, the cleared-defect rate rises from 0.12 to 0.44. The adversary does not need to improve the actual work; it only needs to find a presentation that exploits the judge's reduced scrutiny under high decision load.
Sharding closes this attack surface wherever overload is the cause of the errors. When each criterion receives its own dedicated scrutiny, there is less slack for an adversary to exploit through presentation variation. The defense is strongest precisely when many decisions are competing for attention in the holistic condition.
However, the authors are clear about the limits of this defense. Sharding does not protect against an adversary that persuades the judge on each individual criterion separately, rather than exploiting aggregate overload. For that adaptive attack, the authors find that combining sharding with debate-style opposition within each shard (the S+O configuration) provides stronger resistance.
Theoretical Framing and Practical Implications
Section 7 of the paper offers a capacity interpretation that formalises the intuition. Per-decision attention is decomposed into a budget term and a load term. Increasing compute raises the budget term but leaves the load term unchanged if the number of verdicts stays constant. Sharding lowers the load term while holding both total and per-decision budget fixed. This explains why the two interventions are not interchangeable, even when they cost the same.
The authors also search the space of within-shard communication structures, including opposition, reconciliation, cross-examination, and within-criterion decomposition, and find that all of them are second-order relative to the division itself. Without an active adversary, plain sharding at a calibrated group size is the recommended approach. The optimal group size depends on judge capability, decision difficulty, and the degree of shared context required across criteria.
Several implications follow for evaluation pipeline design:
- Reporting token or tool budgets alone is insufficient. Decision load should be reported alongside budget in any oversight evaluation.
- A weaker sharded judge can outperform a more capable holistic judge, which has real cost implications for evaluation infrastructure.
- Adversarial robustness assessments should include presentation attacks, not just content attacks, particularly for high-stakes review pipelines.
The limitations are worth noting. The persuasion attack is tested primarily in legal review, where the attack is described as most natural. The matched-budget advantage is strongest for the most capable judge tested. And the framework assumes that criteria can be partitioned without losing important cross-criterion context, which will not always hold. Some evaluation tasks require joint reasoning across criteria, and sharding those naively could introduce its own errors.
That said, the empirical grounding across three independent expert-labeled datasets, combined with a clear theoretical account and a well-controlled experimental design, makes this a substantive contribution to the scalable oversight literature. The connection to adversarial robustness is particularly valuable: it reframes what might otherwise look like a mundane engineering choice about how to batch evaluation calls as a security-relevant design decision. For anyone building or auditing LLM-based evaluation pipelines, the decision load variable deserves explicit attention.