Most LLM agents are reactive by design. They observe a state, generate a response, observe the next state, and repeat. This works well enough for short tasks, but it breaks down when decisions made early in a sequence have consequences that only become apparent much later. The paper "Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning" (Zhang et al., 2026) addresses this directly, asking whether an autoregressive language model can be trained to genuinely simulate future states rather than simply pattern-match on what planning-like text looks like.
This is a meaningful distinction. The authors identify what they call a format-capability gap: if you fine-tune an agent on traces that contain look-ahead reasoning, the model learns to produce text that resembles foresight without developing any real predictive grounding. The output looks like planning; the underlying computation is not. Closing that gap is the central contribution of this work.
Key Contributions
The paper proposes a unified training pipeline that treats world-model planning as something that must be built into the model's representations before it can be usefully expressed in its outputs. The three contributions are tightly coupled:
- World Model Agentic Mid-Training (WM-AMT): A mid-training stage designed to inject latent predictive capability into the policy before any structured output format is imposed. The idea is that the model needs to actually learn to predict future states, not just learn a format that gestures at prediction.
- Format-Eliciting SFT (FE-SFT): Supervised fine-tuning applied after WM-AMT, which structures the injected capability into a consistent output format. Crucially, this comes second, not first. The capability exists before the format is elicited.
- Foresight-Conditioned Reinforcement Learning (FC-RL): A reinforcement learning stage that refines the calibration and practical utility of the generated simulations. The agent learns not just to produce plausible rollouts but to produce rollouts that are actually useful for downstream decision-making.
The model is trained to produce two things alongside its actions: a prospective state rollout (a textual simulation of how the environment might evolve) and a plan-conditioned success estimate, which the authors describe as a textual analogue of the Q-value from classical reinforcement learning. Both are generated autoregressively within a single model, avoiding the architectural complexity of separate world-model and policy networks.
Methodology and Training Design
The ordering of training stages is the core methodological bet. Standard practice in LLM post-training is to apply SFT first and RL second. This paper inserts an additional mid-training phase before either, specifically to build predictive representations that SFT can then surface. The authors argue, plausibly, that SFT alone cannot create capabilities that are not latently present; it can only elicit and format what already exists.
The WM-AMT stage is where the most interesting work happens, and also where the paper is necessarily somewhat opaque given the constraints of an arXiv abstract and page limits. The general approach involves training on data that requires the model to predict environmental transitions, grounding the learned representations in actual forward simulation rather than surface-level imitation of planning text.
The FC-RL stage addresses a problem that would arise even with good world-model representations: a model might generate plausible-sounding rollouts that are systematically miscalibrated or that the policy fails to condition on correctly. The RL signal here is tied to whether the generated foresight actually improves decision quality, not just whether it looks reasonable.
Evaluation covers search tasks and mathematical reasoning, two domains with quite different structures. Search tasks have clear environmental dynamics and well-defined success criteria. Mathematical reasoning is more about internal state evolution across reasoning steps. The fact that the approach transfers across both is a reasonable signal that the method is not narrowly overfit to one task type.
Results and What They Show
The paper reports consistent improvements over other training baselines across both evaluation domains. Without access to the full results tables, the key comparison is presumably against: standard SFT on look-ahead traces (which would exhibit the format-capability gap), SFT followed by RL without the WM-AMT stage, and possibly other ordering variants. The consistent outperformance suggests the mid-training stage is doing genuine work rather than being incidental.
The success estimate component is worth noting separately. Producing a textual Q-value analogue is a strong claim. If calibrated correctly, this would mean the model can assess the expected utility of a plan before committing to it, which is qualitatively different from just generating plausible next-state descriptions. Whether the calibration is genuinely reliable or whether the model is producing confident-sounding estimates that happen to correlate with outcomes is something the full paper presumably addresses through ablations.
Limitations and Open Questions
Several questions are not resolvable from the abstract alone but matter for evaluating the contribution:
- The WM-AMT stage requires training data that supports genuine state-transition prediction. The construction and quality of this data is likely the most important practical bottleneck for anyone trying to reproduce or extend this work.
- Autoregressive generation of rollouts is computationally expensive at inference time. The paper does not appear to address the inference cost tradeoff, which matters significantly for deployment.
- The evaluation domains, while usefully different from each other, are both relatively well-structured. It is not clear how the approach would perform in environments with stochastic or partially observable dynamics where accurate state prediction is genuinely difficult.
- The format-capability gap is identified and addressed, but the mechanism by which WM-AMT actually installs predictive representations is not fully explained in the abstract. This is the part of the paper most worth reading carefully.
The broader implication of this work is that the sequencing of training stages matters in ways that the field has not fully worked out. The standard SFT-then-RL pipeline assumes that the capabilities you want to reinforce already exist in some form after SFT. This paper makes a case that for genuinely novel capabilities like world-model planning, an additional capability-injection stage is necessary before structured training can be effective. That framing, if it holds up, has implications well beyond this specific application.
The full paper is available at arXiv:2606.27483.