Catastrophic forgetting remains one of the more stubborn problems in post-training large language models. When you fine-tune a model on a new task, it tends to lose previously acquired capabilities in proportion to how hard it trains on the new objective. Prior work by Shenfeld et al. (2025) established a behavioural observation: RL-based fine-tuning preserves prior capabilities more effectively than supervised fine-tuning (SFT). The explanation offered was that policy-gradient updates stay closer to the base policy, as measured by KL divergence. That is a useful framing, but it leaves open a more fundamental question: what is actually happening inside the model?
This paper, available on arXiv, pushes the analysis to the mechanistic level. Rather than treating forgetting as purely a behavioural or distributional phenomenon, the authors ask whether RL's advantage is reflected in how it handles the internal computational circuits that implement model capabilities. The answer, at least for Qwen2.5-3B-Instruct on scientific question-answering, is a clear yes.
Key Contributions
The central methodological contribution is differential circuit vulnerability, a head-level measure quantifying how much a circuit degrades under a given fine-tuning objective. This allows the authors to move beyond aggregate performance metrics and examine which specific attention heads are preserved, amplified, or weakened by each training method.
The paper also introduces a three-phase experimental pipeline: first reproducing the known behavioural retention gap between SFT and RL; then identifying circuits in each model using Differentiable Binary Masking (DBM); and finally comparing how post-training reshapes those circuits across the two objectives. The code is publicly released at github.com/rl-sft-circuit-research.
Methodology
The experimental setup trains from Qwen2.5-3B-Instruct, applying SFT via completion-only supervision and then refining with Dr.GRPO to obtain the RL model. This means the comparison is not SFT versus RL from scratch but rather SFT versus continued post-training with RL beyond the SFT checkpoint. That design choice isolates the effect of the RL objective specifically, though it also means the two models are not trained on entirely separate trajectories from the base.
Circuit identification uses DBM masking to extract sparse causal subnetworks of attention heads. Faithfulness is measured as the ratio of the circuit's task performance to the full model's performance, with values near 1.0 indicating the circuit captures most of the relevant computation. Head-level shifts are computed by comparing mask values between the fine-tuned models and the base model. Necessity and sufficiency interventions then distinguish heads that are critical bottlenecks from those that contribute in a distributed fashion.
Retention is evaluated across a broad benchmark suite covering commonsense reasoning, factuality, instruction following, and code generation. Using multiple capability domains is sensible here since forgetting is often capability-specific and a single benchmark could give a misleading picture.
Results
The findings are fairly striking in their consistency. DBM identifies a base circuit of 297 attention heads (51.6% of all heads). After fine-tuning:
- SFT compresses to roughly 265 heads (46.0%), retaining about 52% of base-circuit heads
- RL retains approximately 296 heads (51.4%), preserving about 68% of base-circuit heads
- The gap at peak new-task performance reaches 13.5 percentage points (59.0% vs 72.5% after two epochs)
The epoch-level trajectories are particularly telling. SFT's circuit retention declines monotonically across training epochs. RL, by contrast, drops initially and then partially recovers by the second epoch, suggesting the RL objective can consolidate and restore circuit structure rather than simply degrading it more slowly. This non-monotonic behaviour is worth attention: it implies RL training is doing something qualitatively different, not just a gentler version of the same process.
The Differential Causal Mediation (DCM) scores reinforce this picture. RL maintains consistently higher DCM throughout training (15.8 vs 10.4 at Epoch 1; 10.6 vs 6.3 at Epoch 2), indicating that the preserved circuit retains stronger causal engagement with model outputs rather than simply persisting as inert structure.
One result deserves particular notice: the RL model has a larger output-space KL divergence from the base model than the SFT model, yet it preserves more internal circuit structure and retains more prior-task performance. This dissociation between output-space metrics and internal forgetting is a meaningful finding. It suggests that KL divergence from the base policy, while a reasonable proxy, can be misleading as a predictor of internal circuit disruption. A model can shift its output distribution substantially while preserving the underlying computation, or conversely can appear close to the base distribution while aggressively reorganising internal mechanisms.
Implications and Limitations
The authors characterise SFT as a "compressor" and RL as a "distributed adaptor." SFT concentrates new-task behaviour into a smaller, more specialised circuit, discarding much of the base computation. RL spreads adaptation across a broader subgraph, preserving more of the original structure. This framing maps onto the head role analysis: SFT produces a cluster of high-necessity, high-sufficiency critical specialists, while RL maintains a more distributed architecture resembling the base model.
The practical implication is that circuit preservation may be a useful design target for continual adaptation methods. If the goal is to add capabilities without erasing existing ones, then methods that explicitly constrain or regularise circuit disruption could outperform those that rely solely on output-space constraints like KL penalties.
The limitations are acknowledged honestly. The entire analysis rests on a single model (Qwen2.5-3B-Instruct) and a single fine-tuning domain. Whether the same trade-off holds for Llama, Mistral, Gemma, or larger parameter scales is unknown. The circuit analysis is also restricted to attention heads; MLP layers and residual-stream features are not examined, which means the picture is incomplete. It is entirely possible that SFT disruption propagates through MLP components in ways that would change the quantitative conclusions.
There is also a subtlety in the training setup worth flagging. Because the RL model is initialised from the SFT checkpoint rather than trained independently from the base, the comparison conflates the effects of the RL objective with the effects of additional training compute beyond SFT. Disentangling these would require matched compute budgets or training both objectives from the base model independently, which the authors acknowledge as future work.
Despite these constraints, the paper makes a genuine contribution by connecting a well-established behavioural observation to a mechanistic account. The differential circuit vulnerability framework is a concrete tool that others can apply to different models and tasks. The dissociation between KL divergence and internal forgetting is a finding that should inform how the community evaluates and compares fine-tuning methods going forward.