The ARC-AGI benchmark has become something of a proving ground for claims about machine reasoning. Strong results have historically required either frontier models burning hundreds of dollars per task on evolutionary search, or models fine-tuned directly on ARC-distribution data. A new paper, Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1, investigates a third path: can architectural decomposition alone, applied to an open-weight model with no ARC-specific training, close most of that gap at a fraction of the cost?
The answer, at least on the public evaluation set, appears to be yes. Starting from a 15.50% one-shot baseline, the authors reach 67.25% pass@2 at $0.62 per task. That is a 51.75-point lift without touching the model weights or running expensive search procedures. The result matters not just for ARC specifically, but because it sharpens a question the field has been circling: how much of what looks like a capability gap is actually an architectural gap?
Key Contributions
The paper introduces two agentic architectures built on top of DeepSeek V3.2 running in non-thinking mode, served via AtlasCloud FP8 endpoints:
- Explorer-Definer Pipeline: A two-stage system that explicitly separates pattern discovery from transformation synthesis. A PatternExplorer agent analyses training grid pairs and produces structured pattern documents; a TransformationDefiner agent then synthesises executable Python transformations conditioned on those documents. At the canonical configuration (N=5 explorers, temperature 0.5, M=5 definer samples), this reaches 57.50% pass@2 at $0.25 per task.
- Reflective Orchestrator: An augmented version of the definer stage that can autonomously spawn fresh PatternExplorer agents when its current hypotheses fail on training pairs. Rather than committing to a fixed set of pattern documents, the orchestrator can issue
explore_new_patternscalls with structured guidance, effectively looping back upstream when stuck. This reaches 67.25% pass@2 at $0.62 per task.
A third contribution is methodological rather than architectural. The authors use an unbiased pass@k analysis to diagnose whether the pipeline is limited by the quality of its ranking mechanism (selection-bound) or by the diversity of candidates it generates (generation-bound). They find that selection via training-pair accuracy already captures roughly 95% of the candidate ceiling, meaning better ranking would yield almost nothing. The bottleneck is generation diversity. The orchestrator is then framed as a direct test of this prediction: if spawning fresh exploration upstream increases pass@1, the diagnostic was correct. It does, with a +9.8 percentage point unbiased pass@1 lift matching the selection-mediated pass@2 lift.
Methodology
The experimental design is careful about what it controls. All four architectures (no-CoT baseline, structured CoT baseline, pipeline, orchestrator) run on the same model, same provider, same 400-task public evaluation set, and the same cost-accounting code. This makes the within-paper architectural deltas interpretable even if the absolute accuracy numbers do not transfer cleanly to the semi-private leaderboard.
The pipeline sweep covers an N x t x M grid, and the authors report that the Pareto surface is smooth rather than sharply peaked, which partially mitigates the concern that the canonical configuration was cherry-picked from the evaluation set. A cross-model replication on Qwen3-235B-Instruct on a 99-task subset finds the same monotone architectural ordering (baseline to CoT to pipeline to orchestrator), which is the right kind of evidence for generality even if the subset is too small to exclude zero on the Qwen3 orchestrator lift by paired bootstrap.
One ablation worth highlighting: removing the think tool from the pipeline definer reduces pass@2 by 5.75 percentage points. This is a concrete, quantified contribution from within-call deliberation that sits below the level of full chain-of-thought and above zero. It suggests that even lightweight scratchpad access is doing real work in the synthesis stage.
Results in Context
The cost-accuracy positioning is the most striking aspect. Berman's 2024 evolutionary system reaches 53.6% at roughly $29 per task with Sonnet 3.5. This paper's orchestrator reaches 67.25% at $0.62 per task with an open-weight model. The 2025 Berman system reaches 79.6% at $8.42 per task using Grok-4 and natural-language instruction evolution, and Pang's DreamCoder-inspired system reaches 77.1% at an estimated $2.56 per task. So the orchestrator sits below those upper-tier results in accuracy but substantially below them in cost, occupying a different region of the Pareto frontier rather than directly competing.
The three-lever decomposition the paper proposes (within-call deliberation via CoT, across-call decomposition via the pipeline, adaptive re-exploration via the orchestrator) is a clean analytical frame. Each lever is measured independently with paired bootstrap confidence intervals, and each shows a statistically significant positive contribution. The cumulative story, that architecture accounts for most of the gap between a raw one-shot baseline and near-state-of-the-art performance, is well-supported by the data as presented.
Limitations and Open Questions
Several caveats deserve attention. The evaluation is on the public 400-task set, not the semi-private holdout used for official leaderboard scoring. The authors acknowledge this directly and note that any shared public-set bias is held constant across conditions, so the deltas are interpretable even if the absolute level shifts. That is the right framing, but it does mean the headline 67.25% figure cannot be compared to verified leaderboard submissions without qualification.
Configuration selection on the evaluation set is a real methodological limitation. The canonical N=5, t=0.5, M=5 configuration was chosen by inspecting the Pareto surface on the same 400 tasks used for reporting. The authors offer three mitigating arguments: the surface is smooth, the architectural deltas hold across most of the surface, and the cross-model replication preserves the ordering. These are reasonable mitigations but not substitutes for a held-out validation split.
The cross-model replication on Qwen3-235B is encouraging but limited. A 99-task subset is genuinely too small to draw tight conclusions about the orchestrator lift on that model, and the Qwen3 CI touching zero is not just a statistical technicality. Extending this to the full evaluation set and adding a third model family, ideally a non-MoE architecture at a different capability tier, would substantially strengthen the generality claim. The authors identify this as the first item in their future work section, which is the right priority.
The generation-bound diagnostic is one of the more intellectually interesting contributions, and it points directly to where further gains should come from: broader candidate generation upstream rather than better selection downstream. Whether that means more diverse explorer prompting, multi-temperature sampling, or more aggressive spawn budgets in the orchestrator is an open question the current results do not fully resolve.
Overall, this is a well-controlled study that makes a credible empirical case for architectural decomposition as an underexplored axis of ARC progress. The cost positioning is genuinely novel, and the generation-bound diagnostic is a useful methodological contribution that goes beyond simply reporting accuracy numbers. Readers interested in the full details, including the prompt structures, ablation tables, and Pareto sweep results, should consult the full paper on arXiv.