A persistent assumption in multi-agent system design is that adding a dedicated reviewer role should improve outcomes on hard reasoning tasks. Route a candidate answer through a specialist critic, receive a targeted correction, and the solver should do better next time. This paper tests that assumption directly, and the results are uncomfortable for anyone who has built or evaluated hierarchical agent pipelines on that basis. The full paper is available at arXiv:2607.15388.
What the Paper Contributes
The core contribution is a clean empirical separation between two quantities that are easy to conflate: reviewer precision (how often a reviewer warning correctly identifies a real error) and critique uptake (how often a correct critique actually changes the next candidate the protocol submits). The authors introduce three operational metrics to track each step of the correction chain independently: NeglectRate, CouplingRate, and ReviewerGuidedRepairRate. These measure, respectively, how often valid critique is ignored, how often critique propagates into a changed answer, and how often that change actually fixes the problem.
The study compares two multi-agent protocols against a single-agent baseline across 4,181 verifier-grounded problems from Omni-MATH, a ten-tier olympiad benchmark with enough difficulty spread to expose protocol differences. The two protocols are a Planner-Executor-Reviewer (PER) pipeline and a broadcast-style peer discussion in which agents share a common candidate state. All agents use the same model family (gpt-oss-120b), controlling for capability differences between roles.
Methodology and Experimental Design
The experimental design is methodologically careful in ways worth highlighting. Using matched actors across roles eliminates the confound of one role having a stronger underlying model. The ten-tier difficulty structure of Omni-MATH allows the authors to separate regimes where collaboration helps from regimes where it does not, rather than reporting a single aggregate number that obscures both.
The key comparison is between PER and broadcast on the harder tiers. PER has a dedicated reviewer whose critiques are routed back to the executor. Broadcast instead treats the current best candidate as shared mutable state: when a participant proposes a correction, the protocol candidate itself is updated, and subsequent agents review the revised answer rather than the original. The paper includes concrete interaction traces illustrating both failure and success modes, which ground the quantitative claims usefully.
Within-PER interventions probe the uptake hypothesis further. The authors test explicit acknowledgment requirements (ACK-required) and embedding reviewer guidance directly in the solver's working context (EMB). These are not presented as solutions but as diagnostic probes, which is the right framing.
Main Findings
- On tiers 1-2, multi-agent collaboration adds at most around 2 percentage points over the single-agent baseline. The collaboration overhead is essentially wasted on easy problems.
- From tier 4 onward, gains open sharply. On tiers 6-9, broadcast achieves 10-20 percentage point improvements over single-agent, and outperforms PER in final accuracy despite PER having a higher-precision reviewer.
- PER's reviewer precision is 0.861 versus broadcast's 0.644. Yet correct critique in PER is much less likely to change the next submitted candidate. The precision-uptake separation is the central empirical finding.
- ACK-required (forcing explicit acknowledgment of critique) lowers final accuracy rather than improving follow-through. Compliance-style controls on information flow do not substitute for structural coupling.
- EMB (placing reviewer guidance directly in the solver's working context) partially improves follow-through but does not close the accuracy gap with broadcast.
- A cross-family replication with Gemma 3 27B actors preserves the precision-uptake separation: PER's reviewer retains higher precision (0.881 vs. 0.722) while broadcast shows dramatically higher useful-critique uptake (0.742 vs. 0.092 for PER).
The interaction traces are worth examining. In the PER failure case shown, the reviewer generates a concrete corrected answer in the review channel, but the pipeline still submits the earlier candidate. The correction is present as a side-channel comment rather than as active protocol state. In the broadcast success case, a peer correction replaces the shared candidate, which is then re-reviewed and submitted in revised form. The structural difference is not about the quality of the critique; it is about whether the critique becomes part of the live answer trajectory.
Implications and Limitations
The cost analysis adds practical weight to the findings. PER issues roughly 1.6-1.8 times as many verifier calls per problem as broadcast on tiers 4-9, and at tier 10 averages 2,021 seconds per problem versus 519 for broadcast. The scalarized cost-per-solved-problem objective shows that PER only becomes preferable to broadcast when verifier calls are cheap (below roughly 2.1 x 10^5 token-equivalent units per call), and under no tested configuration does PER become cheaper than single-agent on this objective. PER does find a defensible niche under the richer failure-penalty objective on hard tiers, where the cost of a wrong final answer is high but verifier calls are not yet extremely expensive. That is a real deployment scenario, but it requires the right conditions rather than being a general advantage.
The limitations are stated honestly. The evidence is domain-specific: olympiad math has unusually binary correctness signals, and the detection-uptake-repair decomposition may generalise more readily than the exact rate values. Protocol rankings may also shift with model family or prompt configuration, and the cross-family replication, while consistent in direction, is reduced in scale. The authors also note that they cannot yet cleanly distinguish between critique that is not incorporated at all and critique that is seen but treated as less authoritative than the solver's existing reasoning. That distinction matters for system design and is a natural next question.
For evaluation design, the paper makes a pointed observation: a system can score well under reviewer-centric evaluation (high precision, plausible-looking critique) while failing to improve actual problem-solving, because the two quantities are empirically separable. Benchmarking that only measures whether a reviewer identifies errors will miss this failure mode entirely. For system design, the implication is that improving reviewer quality alone is not sufficient; the protocol interface needs to ensure that correct feedback actually changes the next candidate. That sounds obvious stated plainly, but the evidence here suggests it is not how current PER-style pipelines behave in practice.
This is a focused, well-controlled study that identifies a specific and underappreciated failure mode in hierarchical agent pipelines. The detection-uptake-repair decomposition is a concrete methodological contribution that other researchers can apply to their own systems. The finding that broadcast-style shared state outperforms dedicated reviewer routing on hard problems, at lower cost, is worth taking seriously when designing the next generation of reasoning agents.