A persistent concern about using activation steering for alignment is that downstream fine-tuning will simply undo it. If a model provider embeds a safety-relevant behavioural constraint into weights before release, and a downstream user then fine-tunes the model on their own data, does the constraint survive? This paper, available on arXiv, is the first systematic study of that question across multiple models, training paradigms, and steering targets. The answer is nuanced in a way that has direct practical consequences for anyone thinking about embedded steering as a pre-deployment safety tool.
What the Paper Does
The authors embed activation steering directly into model weights for two distinct behavioural targets: refusal suppression (a safety-relevant negative steering case) and brevity induction (a stylistic positive steering case). They then fine-tune five instruction-tuned models ranging from 3B to 14B parameters, Llama-3.2-3B, Llama-3.1-8B, Llama-3-8B, Qwen3-14B, and SOLAR-10.7B, under both supervised fine-tuning (SFT) and RLHF with PPO. Crucially, the training data contains a small but non-trivial fraction of examples that oppose the steered behaviour, roughly 0.1% for refusal-relevant signals in SFT, simulating realistic downstream fine-tuning rather than a targeted adversarial attack.
The study separates two distinct questions that are often conflated. First, does the steered behaviour persist after fine-tuning? Second, does the underlying weight edit persist? These turn out to come apart in a striking way.
Key Findings
- Behavioural preservation is training-content dependent. Refusal ablation loses 64% of its effect on average under SFT, while brevity amplification is largely preserved. Under RLHF, both interventions survive well. The difference tracks the degree of contradictory optimisation pressure in the training data: SFT (using OpenOrca) contains explicit refusal completions to harmful prompts, creating direct pressure against the refusal-suppression steering.
- The weight edit itself is mechanistically stable. Across all models and conditions, fine-tuning reverses at most 0.79% of the embedded edit. Mean vector recovery is reported as ρ = 0.004, and the fine-tuning update along the steering direction is nearly orthogonal to the pre-edit weight pattern (mean cosθ = 0.074). Gradient descent engages the edited direction but does not reconstruct the original weights in a way that cancels the edit.
- Behavioural and mechanistic recovery are largely independent. The correlation between vector recovery and behavioural recovery is r = +0.24 (n = 20, p = 0.31), which is not statistically significant. When steered behaviour degrades, it does so through alternative pathways rather than by dismantling the steering mechanism.
- Embedded steering is therefore mechanistically durable but functionally vulnerable. The weight edit persists, but the model can route around it, rendering it functionally inert without erasing it.
Methodological Considerations
The choice of full-parameter fine-tuning as the experimental setting is deliberate and appropriate: it represents the worst case for steering degradation, since all weights are updated simultaneously. The authors acknowledge that LoRA, which is more common in practice, would likely be less disruptive to the embedded edit due to its low-rank update constraint, though they do not demonstrate this directly.
The experimental design is careful to include contradictory signal without making fine-tuning adversarial. This is a meaningful distinction. The 0.1% refusal examples in SFT are incidental rather than targeted, which is a plausible model of real downstream use. The RLHF setup uses Anthropic hh-rlhf with a reward model that scores compliance with harmful prompts poorly, creating sparser but still opposing signal.
Refusal evaluation uses an LLM judge (DeepSeek-V4-pro) cross-validated against a substring-matching heuristic. This is reasonable, though residual misclassification cannot be fully ruled out. Brevity is measured by mean response length in tokens, which is a clean and interpretable metric for the stylistic intervention.
One genuine limitation is that the paper does not vary the degree of contradictory signal systematically. The relationship between the fraction of opposing examples in training data and the degree of behavioural degradation is inferred from comparing two conditions rather than characterised with a controlled sweep. That would be a valuable follow-up experiment.
Implications and Open Questions
The dissociation between mechanistic and behavioural persistence has several important implications. For open-weight model providers, it means that checking whether the steering direction is still present in the weights after fine-tuning is not a reliable proxy for whether the steered behaviour is still active. The edit can survive while the behaviour reverts, because the model develops alternative circuits that restore the original behaviour without engaging the steering direction. This mirrors deliberate defences against steering attacks described in prior work, and suggests that routine fine-tuning may produce the same effect incidentally.
The finding also has implications for how we think about the causal role of linear representations in model behaviour. The Linear Representation Hypothesis motivates activation steering by positing that concepts are encoded as approximately linear directions in activation space. These results suggest that even when a steering direction is successfully embedded and mechanistically preserved, the model can reorganise how it mediates a given behaviour such that the direction is no longer the operative causal pathway. That is a meaningful qualification of the hypothesis's practical reach.
For practitioners, the paper's recommendations are sensible. Where inference-time hooks are available, they are preferable to embedded steering because the vector can be recalculated after fine-tuning without risk of degradation. Where embedded steering is used, behavioural re-validation after downstream training is necessary, and mechanistic inspection of the weight edit is not a substitute for that. The authors also note that quantisation may pose analogous risks through a different mechanism, which is worth investigating given how commonly quantised models are deployed.
The routing-around phenomenon identified here deserves further mechanistic investigation. The paper establishes that alternative pathways are recruited when behaviour degrades, but does not trace which circuits are involved, whether recovery reuses pre-existing refusal circuits or constructs new ones, or how the recovered mechanism differs structurally from the original. Circuit-level analysis of steered-then-trained models would be a natural and informative extension.
Overall, this paper makes a clean and useful empirical contribution to a question that matters for alignment practice. Embedded steering is not as fragile as the worst-case concern would suggest, but it is not a fire-and-forget mechanism either. The framing of embedded steering as mechanistically durable but functionally vulnerable is a precise and actionable characterisation that should inform how the technique is used and validated in deployment pipelines.
Full paper: arXiv:2608.24988