There is a persistent and somewhat embarrassing result in multi-agent LLM research: carefully orchestrated teams of language models frequently fail to outperform a single well-tuned model. The assumption that more agents equals better reasoning has not held up under rigorous evaluation. This paper, accepted at ICML 2026, offers a principled explanation for why sequential fine-tuning of multi-agent teams is structurally broken, and proposes a concrete fix. The full paper is available at arXiv:2605.15207.
The Core Problem: Compounding Occupancy Shift
When multiple agents share a context and are fine-tuned sequentially, updating agent A changes the distribution of contexts that agent B will subsequently encounter. If agent B's update is then evaluated against rollouts cached before agent A's update, those rollouts no longer reflect the true joint context distribution. The paper formalises this as compounding occupancy shift.
The theoretical contribution here is worth pausing on. The authors prove that under stale-occupancy evaluation, the performance penalty scales quadratically with the number of agents. This is not a minor inefficiency; it means that adding more agents to a sequentially fine-tuned team actively compounds the training error. By contrast, if intermediate-occupancy evaluation is used, the scaling reduces to linear. This distinction provides a clean theoretical basis for why naive sequential fine-tuning degrades as team size grows, and it connects directly to well-understood problems in off-policy reinforcement learning where distribution mismatch between the behaviour policy and the target policy corrupts value estimates.
The framing borrows from trust-region policy optimisation (TRPO and its successors), but extends it to the multi-agent setting in a non-trivial way. Single-agent trust-region methods constrain how far a policy can move from the behaviour policy used to collect training data. TeamTR generalises this by requiring that each agent's update stays within a divergence bound relative to the joint team context distribution, not just its own prior policy.
What TeamTR Actually Does
The proposed framework has three main components:
- Trajectory resampling after each component update. Rather than caching rollouts once at the start of a training stage and reusing them across all agent updates, TeamTR resamples trajectories after each individual agent is updated. This keeps the occupancy estimate fresh and prevents the quadratic penalty from accumulating.
- Per-agent divergence control. Each agent's update is constrained so that the KL divergence between the updated policy and the current team-conditioned distribution stays within a specified bound. This is enforced at the component level, not just at the team level, which is important because team-level constraints can mask individual agents drifting significantly.
- Formal improvement guarantees. The authors derive per-update and per-stage lower bounds on performance improvement. These are not just asymptotic results; they provide concrete guarantees that each update step and each full training stage will not regress, given the constraints are satisfied.
The resampling step is computationally expensive relative to cached-rollout methods, but the paper argues this cost is justified both theoretically and empirically. Whether the overhead is acceptable in practice will depend heavily on the deployment context, particularly when agents are large models with expensive inference.
Experimental Results
The reported results show TeamTR outperforming both single-agent and sequential fine-tuning baselines by 7.1% on average. The paper also reports reduced coordination regressions, which refers to cases where updating one agent causes another agent's effective performance to drop. This is a meaningful metric beyond raw accuracy, since coordination stability matters for deployed multi-agent systems.
A practically useful feature is plug-and-play component replacement: because TeamTR enforces divergence control at the per-agent level, individual agents can be swapped or updated without destabilising the rest of the team. This is relevant for real systems where different agents may be updated on different schedules or replaced with newer model versions.
The experimental section covers reasoning tasks, though the abstract does not detail the specific benchmarks. For a paper of this type, the choice of evaluation tasks matters considerably. Multi-agent gains on structured reasoning benchmarks do not always transfer to open-ended generation or dialogue tasks, and it would be worth examining whether the improvement holds across diverse task types.
Limitations and Open Questions
Several questions are worth raising for readers considering building on this work.
First, the resampling requirement means TeamTR is substantially more expensive than sequential fine-tuning. For large models, resampling after every component update could multiply inference costs by a factor proportional to team size. The paper is 9 pages, which is tight for a full treatment of computational overhead, and it is unclear how the method scales to teams of, say, 10 or more agents.
Second, the theoretical guarantees assume that the divergence constraints are actually enforceable during training. In practice, KL divergence constraints on language model outputs are approximated, often via clipped importance ratios as in PPO. The gap between the theoretical framework and the practical implementation deserves more scrutiny.
Third, the paper focuses on shared-context teams, which is one specific architectural pattern for multi-agent LLM systems. Architectures where agents communicate via discrete messages or operate over separate context windows may have different failure modes, and it is not obvious that the compounding occupancy shift analysis applies directly.
The broader implication is significant. If the quadratic penalty result holds generally, it suggests that much of the existing literature on multi-agent LLM fine-tuning has been inadvertently training under conditions that guarantee performance degradation at scale. TeamTR offers a principled correction, and the formal guarantees give it a stronger theoretical foundation than most work in this space. Whether the community adopts resampling-based approaches will likely depend on whether the computational cost can be reduced, perhaps through importance weighting approximations that avoid full trajectory resampling while preserving the linear rather than quadratic scaling.
For researchers working on multi-agent coordination or RLHF-style fine-tuning of LLM ensembles, this paper is worth reading carefully. The theoretical framework is clean, the identified failure mode is real, and the proposed solution is grounded rather than heuristic. Full details at arXiv:2605.15207.