Most autonomous optimisation systems are built around a comfortable assumption: the thing being optimised is self-contained. Tune a kernel, improve a schedule, rewrite an attention operator. The problem is that real production software does not work this way. Fixing a dispatch path in an attention module can expose a missing shape-specific kernel, which then triggers a resource-pressure regression in the serving layer under load. Each layer is invisible until the one above it is resolved, and each demands different expertise to fix. Arbor is a framework that takes this compositional structure seriously, and the results suggest that doing so matters considerably more than the field has acknowledged.
What Problem Arbor Actually Solves
Prior work on LLM-guided code optimisation, including AlphaEvolve, AVO, and multi-agent kernel synthesis systems like KernelSkill and AccelOpt, has produced impressive results on isolated computational targets. These systems treat evaluation as stateless: try a candidate, measure it, move on. That approach works when the action space is flat and interventions are independent. It breaks down when optimisations interact across stack layers, when failures carry diagnostic information that should constrain subsequent search, and when a successful intervention shifts the bottleneck to a previously invisible part of the system.
The core reframing in Arbor is treating cross-layer performance optimisation as heuristic tree search over a dynamically expanding action space, where the tree itself serves as shared working memory across agents. Each node encodes an action and its outcome (kept, reverted, or crashed), and the path from root to any node defines the cumulative configuration state. Crucially, the tree is not fixed at initialisation. After a successful intervention, re-profiling can reveal bottlenecks that did not exist before, generating branches that expand the frontier into previously nonexistent regions of the action space.
This is a meaningful departure from how autonomous optimisation has been framed. The challenge is not just generating good candidates; it is navigating a stateful space where each action reshapes what subsequent actions are possible or worthwhile.
Architecture and Methodology
Arbor's multi-agent architecture is built around an explicit tension between two competing pressures. The Orchestrator drives aggressive optimisation by profiling the system, scoring candidate interventions, and dynamically constructing Domain Specialist agents for each task. The Critic monitors the full optimisation trajectory and enforces stability constraints. Neither agent can act unilaterally: the Orchestrator cannot retain changes the Critic flags as unstable, and the Critic cannot block exploration without diagnostic evidence.
Several design choices here are worth examining carefully:
- Dynamic specialist construction. Domain Specialists are not pre-defined agents with fixed prompts. The Orchestrator assembles each specialist at runtime by composing the task mandate with hardware context, retrieved knowledge base entries, and the history of prior attempts on similar targets. The same conceptual "kernel specialist" receives different domain knowledge depending on the model and platform. This is a practical solution to the context management problem in long-horizon campaigns.
- Failure as signal, not noise. Crashes trigger root-cause analysis that converts failure mechanisms into constraints on child nodes. Regressions trigger introspection that evaluates whether the underlying idea retains merit under a different implementation. This is the mechanism that allows the system to distinguish genuinely unproductive directions from implementation errors.
- Hard and soft skills. Agent capabilities are decomposed into domain expertise (hard skills) and coordination protocols (soft skills): when to delegate, how to incorporate another agent's findings, when to escalate. This decomposition is what allows multi-day campaigns to remain coherent as context windows fill and the event history grows.
- Persistent knowledge base. Strategies and failure modes accumulate across sessions, enabling warm-start transfer to new models or hardware generations without repeating known-bad directions.
The search procedure itself is depth-first with dynamic re-scoring after each measurement. Every candidate is gated through accuracy verification before updating the tree, since local metric improvements can regress the full system in ways that only appear at end-to-end evaluation.
Results
Arbor is evaluated on six production models served on AMD Instinct MI355X GPUs, with baselines drawn from InferenceX configurations reflecting extensive manual engineering effort. At concurrency 64, throughput improvements over the vendor baseline are:
- gpt-oss-120b (MoE 120B, MXFP4): +48%
- DeepSeek-R1-0528 (FP8): +90%
- MiniMax-M2.5 (FP8): +50%
- GLM-5-FP8 (FP8): +193%
- Qwen3.5-397B-A17B (FP8): +40%
- Kimi-K2.5 (MXFP4): +60%
The GLM-5-FP8 result is the most instructive. Reducing tensor parallelism from TP=8 to TP=4 while co-optimising the NSA attention kernel and MoE dispatch yielded a 2.9x throughput improvement. No single-layer optimisation could have found this; it required recognising that the interaction between parallelism degree and kernel efficiency was the actual bottleneck. The DeepSeek result similarly reflects compounding across MTP speculative decoding, CK GEMM kernel rewrites, fused allreduce, and configuration tuning discovered across the campaign.
The ablation comparison is stark. A single agent without the Arbor harness plateaus at +33% throughput improvement and crashes irrecoverably within hours. The full system sustains multi-day campaigns and achieves results that are qualitatively different in character, not just incrementally better.
Cross-generation validation on MI300X (+62-99%) and independent replications converging within 2 percentage points confirm that the approach is hardware-agnostic and reproducible, which matters for claims about production deployability.
Limitations and Open Questions
The paper acknowledges limitations without fully quantifying them, which is worth noting. The computational cost of running frontier-tier reasoning models (at the capability level of Claude Opus 4.6/4.7) as the backbone for all agents across multi-day campaigns is not discussed in detail. For organisations without access to such models, or operating under tight inference budgets, the economics of the approach are unclear.
The evaluation domain is also narrow by design. LLM inference serving on AMD Instinct GPUs is a well-instrumented, measurable environment with clear performance metrics and established profiling tooling. How well the tree search formulation transfers to domains where the feedback signal is noisier, slower, or harder to interpret remains an open question. The paper frames Arbor as domain-agnostic in principle, but the evidence base is domain-specific.
There is also a question about the Critic's constraint mechanism. The paper describes the Critic as unable to block exploration without diagnostic evidence, but the quality of that evidence depends on the LLM's root-cause analysis capabilities. In failure modes that are subtle or involve hardware-specific behaviour outside the training distribution, it is not obvious how reliably the Critic distinguishes genuinely unproductive directions from cases where better implementation would succeed.
These are real limitations, but they do not undermine the core contribution. The framing of sustained autonomous optimisation as a coordination and state management problem, rather than purely a search or generation problem, is the right framing. The checks-and-balances architecture is a concrete instantiation of that insight, and the results demonstrate it works at production scale. For anyone building autonomous systems that need to operate in complex, stateful environments over extended time horizons, Arbor is worth reading carefully.
Full paper: arXiv:2606.12563