The post-training alignment of large language models is one of the most consequential steps in modern AI development, yet until recently it has been evaluated almost exclusively through behavioural benchmarks. We know that PPO, DPO, and their descendants change what models say; we know surprisingly little about how they change the internal computations that produce those outputs. This paper, available at arXiv:2606.09850, addresses that gap directly by applying a suite of mechanistic interpretability tools to six alignment objectives across three open-weight model families.
The stakes here extend well beyond academic curiosity. If two alignment methods produce similar behavioural scores but radically different internal representations, their safety properties, generalisation behaviour, and susceptibility to adversarial pressure may diverge in ways that benchmark numbers cannot detect. A white-box account of alignment is therefore a prerequisite for any serious interpretability-based safety auditing.
What the Study Does
The authors train aligned variants of Llama-3.2-3B-Instruct, SmolLM3-3B, and Qwen3-4B-Instruct using PPO, DPO, SimPO, ORPO, KTO, and GRPO, all on the UltraFeedback dataset with a consistent LoRA configuration. They then apply three complementary diagnostic tools to the resulting models:
- Layer-wise linear probing. Logistic regression classifiers trained on residual-stream activations at each layer, predicting whether a response is chosen or rejected. This identifies where preference information is most linearly decodable and how alignment shifts that signal.
- Sparse Autoencoders (SAEs). Trained at the probe-optimal layer to decompose superposed activations into sparse, interpretable feature directions. This allows the authors to identify which specific features carry preference-relevant information.
- Crosscoders. Joint encoders trained on base and aligned model activations simultaneously, enabling direct comparison of shared, amplified, attenuated, and model-specific features. This is the most structurally informative tool for characterising how alignment reshapes the latent space.
The combination is well chosen. Linear probing is fast and provides a global separability signal but is non-causal. SAEs give feature-level decomposition but require careful training. Crosscoders are the most powerful tool for model comparison but are also the most expensive and the least standardised. Using all three together allows the authors to triangulate findings that no single method could establish alone.
Main Findings
The central result is that alignment algorithms cluster into three groups based on their effect on internal preference representations:
- KTO and GRPO consistently improve the linear decodability of preference signals. Probe accuracy for KTO on Llama-3.2-3B reaches 0.958, compared to 0.844 for the base model. GRPO achieves 0.946 on the same model. Crosscoder analysis attributes this to constructive feature sharing and the recruitment of sparse, high-salience feature directions.
- DPO and ORPO reduce linear separability relative to the base model on Llama-3.2-3B and Qwen3-4B. DPO probe accuracy drops from 0.844 to 0.803 on Llama-3.2-3B; ORPO drops to 0.786. The mechanisms differ: DPO is associated with non-constructive rotation of shared features, while ORPO attenuates the activation magnitude of preference-relevant directions without substantially rotating them.
- PPO and SimPO largely preserve baseline geometry. Their probe metrics sit close to the base model across all three architectures, suggesting these objectives do not substantially restructure the internal preference representation even when they improve behavioural outputs.
The ORPO result on Llama-3.2-3B is particularly striking. Its MT-Bench average score collapses to 2.12 against a base of 6.33, which is consistent with the internal picture of severe feature attenuation. The fact that the mechanistic analysis and the behavioural benchmark agree here is encouraging for the validity of the approach, though the authors do not dwell on this alignment between the two evaluation modes.
Preference representations consistently concentrate in early-to-mid or mid-to-late layers depending on the model family, but the specific layer varies by alignment method. This means that probing a single fixed layer across methods would give a misleading picture; the layer-wise sweep is methodologically important.
Architecture Dependence
One of the more practically significant findings is that the same alignment objective can produce qualitatively different internal effects depending on the base model. SmolLM3-3B shows a different pattern for ORPO and GRPO than Llama-3.2-3B does. KTO produces a large increase in probe accuracy on SmolLM3-3B (0.922) but a more modest effect on Qwen3-4B (0.970, though from a higher base). GRPO's effect on the anchor feature in SmolLM3-3B appears at a different layer than the probe-optimal layer, suggesting the method's internal footprint is somewhat displaced relative to the other architectures.
This architecture dependence is not surprising given that pretraining data, tokenisation, and attention patterns differ substantially across families, but it has an important practical implication: claims about the internal effects of a given alignment method cannot be straightforwardly generalised across model families. Auditing the internal representations of an aligned model requires model-specific analysis, not just method-specific assumptions.
Limitations and Open Questions
The authors are appropriately candid about the scope of their claims. The study is limited to 3B-4B parameter models, and it is genuinely unclear whether the geometric patterns observed here scale to 70B+ models where feature superposition density and redundancy may behave quite differently. The analysis is also diagnostic rather than causal at the component level. The probing and SAE results tell us where preference information lives and how its distribution changes, but they do not establish which attention heads or MLP layers are necessary or sufficient for the observed behaviour. Activation patching and path patching experiments would be needed to close that gap.
There is also a question the paper does not fully address: what does reduced linear separability in DPO and ORPO actually mean for safety? The prior work by Lee et al. (2024) showed that DPO reduces linear decodability of toxic features in early layers while shifting them to later layers. If preference representations are being rotated rather than erased, they may still be recoverable through non-linear probing or fine-tuning attacks. The crosscoder analysis provides some evidence on this, but a more direct test of recoverability would strengthen the safety implications considerably.
The use of LoRA rather than full fine-tuning is a practical choice that may itself introduce confounds. LoRA constrains the update to a low-rank subspace of the weight matrices, which could plausibly explain why some methods preserve baseline geometry more than others if their optimal update directions happen to align poorly with the LoRA subspace.
Why This Matters
The paper makes a credible case that alignment is a heterogeneous intervention and that the field's current reliance on behavioural benchmarks is insufficient for understanding what alignment actually does to a model. The finding that KTO and GRPO enhance preference separability through constructive mechanisms while DPO and ORPO degrade it through geometric distortion has direct implications for how we should think about interpretability-based safety monitoring: a probe trained on a DPO-aligned model may be measuring something quite different from one trained on a KTO-aligned model, even if both models produce similarly aligned outputs.
The call for standardised feature-level auditing protocols is well motivated by these results. As alignment methods proliferate and models are deployed in higher-stakes settings, the ability to verify that a model's internal representations are consistent with its stated alignment properties becomes increasingly important. This paper provides a useful methodological template for that kind of auditing, even if the template will need to be extended to larger models and causal analysis before it can be applied with confidence in production settings.
Full paper: arXiv:2606.09850