The claim that language models can develop broadly misaligned behaviour through narrow fine-tuning on domain-specific data has attracted significant attention in AI safety research. The phenomenon, termed Emergent Misalignment (EM), was reported in work by Betley et al. (2025) and Turner et al. (2025), and raised a troubling possibility: that fine-tuning on seemingly innocuous but subtly misaligned data could produce sweeping behavioural changes far beyond the training distribution. A new preprint, "An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?", systematically re-examines these claims and finds the evidence considerably weaker than previously suggested.
This matters because the EM framing has begun shaping how researchers think about alignment fragility and the risks of fine-tuning deployed models. If the phenomenon is real and robust, it represents a serious practical threat. If it is largely an artefact of experimental design, then the field risks misallocating research effort and overstating certain threat models.
Key Contributions
The paper makes three distinct contributions worth separating out:
- A systematic replication study using controlled align-misalign-align fine-tuning loops, probing both "bad-good-bad" and "good-bad-good" training orders on Qwen2.5-14B-Instruct.
- An identification of response-length confounding as a major driver of apparent rapid realignment, which disappears once dataset distributions are normalised for token length.
- A mechanistic analysis tracking LoRA adapter representations across checkpoints, finding that previously reported representational phase transitions do not consistently correlate with behavioural misalignment.
The authors work within a "model organism" setup inherited from Turner et al., using parameter-efficient fine-tuning via LoRA adapters with frozen base weights. This is a methodologically sensible choice: it constrains the experimental surface while allowing controlled variation across adapter configurations (rank-1 versus rank-32) and training orders.
Methodology
The core experimental design involves cycling a pre-aligned instruction-tuned model through alternating fine-tuning phases using a paired risky financial advice dataset, containing matched safe and unsafe responses for the same prompts. This pairing is important because it allows the training signal to be varied while keeping the semantic domain constant.
Behavioural evaluation follows the Betley et al. protocol: a holdout set of eight benign questions, 50 sampled responses each at temperature 1, scored by GPT-4o-mini on alignment (0-100) and coherency (0-100). A response is classified as emergently misaligned if alignment scores fall below 30 and coherency remains above 50. Checkpoints are evaluated every 25 steps throughout training.
The mechanistic analysis tracks cosine similarity between successive LoRA A and B matrix checkpoints across attention layers, aiming to detect abrupt directional changes in parameter space that might correspond to misalignment events. This approach is reasonable as a first-pass signal, though it is worth noting that cosine similarity between consecutive adapter states is a fairly coarse measure of representational change.
One methodological detail deserves attention: the authors extend the original Turner et al. dataset by generating paired safe financial advice using GPT-4o, producing 6,000 paired examples. The choice to focus exclusively on this dataset, justified by its producing the strongest misalignment signal in prior work, is pragmatically sensible but does limit generalisability of the findings across domains. The authors do briefly report consistent trends with a bad medical advice dataset, which partially addresses this concern.
Results and What They Actually Show
The headline finding is that apparent rapid realignment, observed initially in both single-adapter and all-adapter configurations, largely disappears once response lengths are controlled. In the unnormalised setting, as few as 40 safe training examples were sufficient to dramatically suppress misalignment rates. This looks, superficially, like strong evidence that EM is highly surface-level and easily reversed.
The problem is that the safe financial advice dataset contained systematically longer responses than the unsafe version. After normalising token lengths while preserving semantic content, the model becomes susceptible to repeated misalignment and realignment cycles again, suggesting the length difference was acting as a confounding signal rather than genuine alignment information. This is a clean and important result. It demonstrates that evaluation conclusions about EM reversibility can be driven by distributional artefacts that have nothing to do with alignment per se.
The mechanistic results are more ambiguous. Unlike Turner et al., the authors do not observe a reproducible representational phase transition in LoRA space that reliably precedes or coincides with behavioural misalignment. Cosine similarity profiles show oscillatory drift throughout training regardless of alignment direction. This is a negative result, but an informative one: it suggests that the mechanistic signatures reported in prior work may not generalise across model scales, adapter configurations, or training regimes.
The StrongReject jailbreaking benchmark showed no strong misalignment signal, which the authors attribute to EM being specifically about harmful responses to benign questions rather than standard jailbreak-style misalignment. This distinction is worth preserving carefully in future work, as conflating the two phenomena risks muddying both literatures.
Limitations and Open Questions
Several limitations are worth flagging. The study uses a single base model (Qwen2.5-14B-Instruct) and a single primary dataset domain. Whether these findings generalise to other model families, scales, or misalignment datasets remains open. The reliance on GPT-4o-mini as an LLM judge introduces its own calibration uncertainties, a problem the authors acknowledge in passing when noting that alignment lacks a continuous, well-calibrated metric.
The mechanistic analysis, while a genuine contribution, is limited by the choice of cosine similarity between consecutive checkpoints as the primary signal. More informative approaches might include probing classifiers trained on intermediate representations, or tracking the evolution of specific attention heads known to be relevant to refusal behaviour.
The paper also does not fully resolve whether EM is absent or simply harder to demonstrate robustly. The authors are careful to make this distinction, but it is easy to read the results as a debunking when the more accurate framing is that existing evidence is insufficient to establish the phenomenon's robustness. These are different claims with different implications for the field.
For researchers working on alignment evaluation methodology, fine-tuning safety, or mechanistic interpretability of aligned models, this paper is worth reading carefully. The response-length confound in particular is the kind of finding that should prompt re-examination of prior experimental setups. It is a reminder that alignment evaluation, lacking the continuous and calibrated metrics available in other ML subfields, remains particularly vulnerable to subtle distributional artefacts shaping conclusions in ways that are not immediately obvious.
The full paper is available at arxiv.org/abs/2607.09053.