Most work on improving frozen large language models as agents treats the surrounding scaffolding as a single string to be optimized in bulk. A new paper, arXiv:2609.02889, asks a sharper question: which part of that scaffolding actually carries the optimization value? The answer turns out to matter a great deal, not just for understanding why prompt evolution works, but for avoiding a failure mode the authors call the budget-splitting trap, where structuring the optimization actually makes things worse.
The Problem With Flat Harness Optimization
Reflective prompt evolution methods have shown that iteratively mutating and scoring an agent's instruction string can substantially improve task performance without touching model weights. The appeal is obvious: black-box access, no gradient computation, and competitive results against reinforcement learning on benchmarks like ALFWorld. But these methods optimize the harness as an undifferentiated whole. A persona line, a task-decomposition strategy, an action-format rule, and a loop-control heuristic all get swept into the same string and mutated together. When the method works, you know the harness improved; you have no idea which component was responsible.
This is a meaningful gap. Budget allocation, interpretability, and transfer across tasks all depend on knowing whether the gain came from the persona, the strategy specification, the format constraints, or the control logic. Without that map, every new harness-optimization effort starts blind.
HarnessEvo: A Structured Microscope
The paper introduces HarnessEvo, which decomposes the agent harness into four named, separately evolvable slots: role/persona (c1), task-strategy (c2), tool/format-rules (c3), and reflection/control-heuristics (c4). These are evolved one at a time by coordinate ascent, each using the same underlying reflective optimizer (GEPA), under a fixed total rollout budget equal to what flat single-string evolution receives. The key methodological addition is a leave-one-in / leave-one-out (loi/loo) credit-assignment protocol. A leave-one-in run evolves a single slot in isolation and measures its marginal gain over the stock harness. A leave-one-out run evolves all slots except one and measures how much the full harness degrades without it. Together these give a per-slot credit map on held-out tasks, with paired McNemar significance tests.
The authors are explicit that HarnessEvo is a microscope, not a better agent. They do not claim the decomposition yields superior performance. They claim it reveals something the flat methods cannot see.
Main Findings
On ALFWorld with a frozen 7B backbone (Qwen2.5-7B), the top-line result looks unremarkable. HarnessEvo-full reaches 0.657 held-out success against 0.642 for both the stock harness and flat-string evolution. Neither difference is significant (McNemar p = 0.617 and p = 0.480 respectively). A reader stopping at the headline number would conclude that structured harness evolution does nothing.
The loi map tells a different story:
- Evolving only the reflection/control slot (c4) lifts held-out success by +0.119 (p = 0.0046; 22 vs 6 discordant tasks in favour of the evolved control).
- Evolving only the role, strategy, or format slot yields gains of +0.030, +0.007, and +0.007 respectively, none of which approach significance.
- The full structured method gains only +0.015, while the sum of the four individual loi gains is +0.164, an order of magnitude larger.
This strong sub-additivity is not mysterious once you understand the optimizer's mechanics. The reflective optimizer has an accept-and-rescore floor: it needs a minimum number of rollouts per slot to score a parent, evaluate a mutant, and re-score an accepted child on the full validation set. Splitting a budget of 64 rollouts evenly across four slots leaves 16 per slot, which is below that floor. Every slot froze at its empty seed. The structured method produced a harness byte-identical to stock and never captured c4's gain at all.
The practical implication is direct. Concentrating the full budget on the high-credit slot recovers what the split discards. A targeted control-only run at 32 rollouts reaches 0.761 (+0.119, p = 0.0046). An all-to-control run at the full 64 rollouts reaches 0.724 (+0.082, borderline at p = 0.0725). The authors report both results honestly and explicitly decline to overclaim at the larger budget given the noise at n = 134. The directional conclusion is clear: targeted allocation at even half the split budget outperforms uniform splitting at the full budget.
On WebShop the picture is different in an informative way. Every slot froze empty regardless of budget concentration, and a targeted control-only run also stayed null. This rules out budget starvation as the explanation on WebShop. The optimizer had concentrated budget and still found nothing to accept. The authors attribute this to a genuine task-level property: ALFWorld exhibits recurrent, verbalizable control failures (repeated dead-end actions, closed receptacles, premature stopping) that a general heuristic can address, while WebShop's attribute-match purchasing task does not have a control failure mode that a fixed heuristic can fix. The value that harness optimization captures, where it captures anything, is environment-grounded self-correction, not reward-exploiting strategy.
Implications and Limitations
The budget-splitting trap is a real and underappreciated failure mode. It arises from an interaction between a seemingly sensible design choice (structured decomposition with equal budget allocation) and a structural property of the optimizer (a minimum rollout floor for accepting mutations). Any practitioner building structured prompt-evolution pipelines should check whether their optimizer has an analogous floor before splitting budget uniformly across components.
The localization finding also has broader significance. It is consistent with work on unit importance in neural networks, where single directions or components can carry disproportionate functional weight. The analogy here is that one of four harness slots carries essentially all the optimization value, and the other three are close to redundant with what the frozen model already knows. This connects to the framing of in-context learning as implicit optimization: slots that restate capabilities the model already has (persona, format) add nothing, while a slot that supplies missing control structure does.
The limitations are clearly stated. Results come from a single 7B backbone; whether localization persists, shifts, or disappears at larger scale is explicitly flagged as unknown. Two benchmarks cannot map the full space of when harness optimization helps. The coordinate-ascent scheme with one specific optimizer defines the sub-additivity mechanism, and other decompositions or optimizers might behave differently. Sample sizes are modest, which is why the budget-allocation effect is significant at 32 rollouts but only borderline at 64.
The most useful near-term directions the paper identifies are an automatic slot-credit estimator that allocates budget proportionally to estimated per-slot value, and a taxonomy of task types by their control-failure profile to predict a priori whether targeted harness optimization will pay off. Both would turn the manual prescription demonstrated here into a practical one-pass pipeline.
For anyone working on prompt optimization, agent scaffolding, or the broader question of where LLM agent performance comes from, this paper is worth reading carefully. The headline result is a tie, but the mechanism it uncovers is not. Full details at arXiv:2609.02889.