Most work on improving LLM agents without retraining follows a familiar recipe: freeze the model weights and inject an evolving natural-language artifact into the context. The artifact might be accumulated reflections, induced workflows, a curated playbook, or an optimized system prompt. Papers in this space almost universally report gains, but almost always on a single benchmark, against weak or mismatched baselines, with a backbone chosen to flatter the method. The result is a literature that is hard to read and harder to trust.
This paper, Recursive Self-Evolving Agents via Held-Out Selection, attempts a corrective. The authors run six representative context-evolution methods on a single shared backbone across four diverse benchmarks, and the picture that emerges is considerably less flattering than the individual papers suggest. More importantly, they identify the mechanism that actually governs reliability: not the form of the artifact, but whether a held-out selection gate decides whether to commit it.
What RSEA Does Differently
The proposed method, RSEA (Recursive Self-Evolving Agent), maintains a three-layer natural-language state: a high-level imperative strategy, a library of reusable skills, and a playbook of distilled procedures from successful trajectories. Across generations, the agent rolls out its current state on a small evolve pool, rewrites all three layers from the resulting trajectories, then commits the candidate only if it does not regress on a disjoint held-out validation split. The frozen best state is what gets deployed at test time, so there is no leakage.
The three-layer decomposition is motivated by a real architectural complaint about prior work: methods like AWM, ACE, and Dynamic Cheatsheet fold qualitatively different kinds of knowledge into a single string. A high-level heuristic about how to approach a task, a reusable sub-routine for object manipulation, and a concrete step-by-step procedure for a specific task type are not interchangeable. Conflating them into one artifact forces the model to do implicit disambiguation at inference time, which is exactly the kind of thing that causes context distraction.
The retry variant, RSEA-R, injects the frozen evolved prior into a per-task retry loop, combining the cross-task generalisation of the evolved state with within-task adaptation.
The Comparative Study
The six baselines are ReAct, Reflexion, GEPA, AWM, ACE, and Dynamic Cheatsheet, all re-implemented on the same local backbone (Qwen2.5-7B for ALFWorld, 30B for the transfer benchmarks). The four benchmarks span household task completion (ALFWorld), general-purpose web-assisted QA (GAIA), tool-use dialogue (tau-bench), and product search (WebShop). This is a genuinely diverse set, and the diversity is what makes the results interesting.
Key findings:
- No artifact universally wins. RSEA is the strongest single-pass method on ALFWorld (69.3% vs. ReAct 64.6%, McNemar p=0.015) and RSEA-R achieves the best overall result (79.4%). On the strong-backbone tool-use benchmarks, AWM's concrete workflow induction provides the only consistent small lift; RSEA ties ReAct and never significantly regresses.
- Unguarded context evolution is genuinely dangerous. Dynamic Cheatsheet, which curates context online with no held-out gate, scores 70.7% on ALFWorld, near the top of the single-pass column, but collapses to a WebShop score of 0.136 against ReAct's 0.429. This is not a marginal regression; it is a catastrophic failure caused entirely by the context the method added.
- The held-out gate is what makes evolution safe. Ablations show that removing held-out selection causes severe overfitting: perfect in-sample performance but a 33-point drop to test. Every layer of the three-layer state contributes, with the layers partially overlapping rather than being strictly complementary.
Why the Selection Gate Matters More Than the Artifact
The central argument is worth unpacking carefully. An evolution loop that commits whatever it produces inherits the full variance of its own rewrite process. When the backbone is strong and the task is well-suited to the artifact form, this works fine. When it is not, the injected context actively degrades the base policy. The authors call this context distraction, and it is a more precise framing than the usual vague concern about context length or noise.
A strict keep-better gate over a held-out split converts this from a coin-flip into a monotone-safe operation: the evolved agent cannot do worse in expectation than the base agent, because it falls back to vanilla ReAct whenever the evolved context would hurt. This is a clean theoretical property, and the empirical results bear it out. RSEA never significantly underperforms ReAct on any benchmark tested, which is a stronger reliability claim than any of the baselines can make.
The comparison to ACE is instructive here. ACE uses a val keep-best rule, but the authors classify it as non-strict because it uses add-only deltas rather than a holistic rewrite evaluated against a disjoint split. The distinction matters in practice: add-only policies can accumulate context that is individually non-harmful but collectively distracting, and they cannot recover from a bad early addition.
Limitations and Open Questions
The authors are candid about scope. The transfer benchmarks use a single seed of the shuffled split, and GAIA's live web retrieval adds run-to-run noise that can flip a one-task gap. The strict gate guarantees safety on held-out validation, not on every test draw when the validation split is small, which is a real caveat for low-data settings. The entire study is confined to the weight-frozen, NL-state regime; comparisons to code-updating or weight-updating self-improvement are out of scope.
There are also questions the paper does not fully address. The three-layer decomposition is motivated intuitively, but the ablations show the layers overlap rather than being strictly complementary. It is not obvious how to choose the decomposition for a new domain, or whether the specific three-layer form generalises beyond the benchmarks tested. The held-out gate is also only as good as the validation split is representative; in highly non-stationary environments, a gate calibrated on one distribution could still commit harmful context for a shifted test distribution.
The broader implication for the field is methodological as much as technical. The paper's most useful contribution may be the shared harness and pre-registered manifests, which provide a template for honest comparison in a subfield that badly needs one. The finding that no artifact universally wins is not a pessimistic result; it is an accurate one, and it points toward the right research question: not which artifact is best, but under what conditions does any artifact help, and how do you know before committing it.
The full paper, baseline re-implementations, and evaluation harness are available at arxiv.org/abs/2606.28374.