A persistent question in the reasoning distillation literature is whether the proliferation of offline RL loss functions, each accompanied by claims of superiority, actually produces mechanistically distinct models. Benchmark accuracy alone cannot answer this: two methods could reach the same score via entirely different weight updates, or reach different scores while modifying the same subspace. This paper, available at arXiv:2606.23740, takes a different approach by analysing the weight-space geometry of six offline training methods directly, using cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA.
The practical stakes are real. Practitioners choosing between SFT, RFT, RIFT, DFT, Offline GRPO, and DPO are often doing so on the basis of a few benchmark numbers, without knowing whether they are selecting genuinely different inductive biases or just different hyperparameter conventions. Interpretability researchers face the same ambiguity: does "offline RL fine-tuning" name a single mechanistic family or several?
Methodology
The experimental design is notably controlled. All six methods train on identical math rollouts generated from a single base model, Qwen3-4B-Instruct, with attention-only LoRA and a shared initialisation. The analysis then operates on the LoRA deltas directly, treating each method's weight update as a geometric object in the same ambient space.
The four analysis tools each probe a different aspect of the geometry. Global and per-layer cosine similarity measures directional alignment. Per-layer SVD and principal-angle analysis between top-k singular vector subspaces captures whether two methods adapt the same low-dimensional directions, even if their magnitudes differ. Linear mode connectivity interpolates between two weight deltas and measures the loss barrier along that path, a signature of whether two solutions sit in the same loss basin. CKA on hidden states provides a representation-level complement to the weight-level analysis.
One important caveat the authors are transparent about: DPO uses a 10x smaller learning rate than the other five methods, following standard codebase conventions. This means DPO's update norm and accuracy numbers cannot be cleanly attributed to loss formulation alone. A learning-rate-matched comparison is flagged as future work.
Key Findings
The results fall into four distinct groups:
- SFT, RFT, and RIFT are geometrically near-identical. Pairwise cosine similarities reach 0.97 or above, and median top-1 principal angles between their subspaces are 6.7 and 8.2 degrees respectively. GSM8K accuracy is statistically indistinguishable across this cluster (87-88%, McNemar p >= 0.15, n=1319). RIFT's reward-weighted formulation, which was designed to incorporate negative samples, produces essentially the same weight update as plain SFT on this data.
- DFT diverges more than any explicitly reward-weighted method. Despite being a one-line modification of SFT that down-weights confident tokens using a stop-gradient, its median top-1 principal angle from SFT is 26.7 degrees, larger than Offline GRPO's 18.5 degrees. This is a counterintuitive result: a structural change to how the loss weights individual tokens has more geometric impact than adding explicit reward signals.
- Offline GRPO adds a substantial orthogonal component. Globally, 67% of its update is orthogonal to the SFT direction, rising to 79-86% in the final five transformer blocks. Yet linear mode connectivity shows it remains in the same loss basin as SFT and RIFT, suggesting it extends rather than redirects the SFT update.
- DPO occupies a near-orthogonal subspace. The median top-1 principal angle from SFT is 54.6 degrees, and the IQR for the worst (top-10) principal angle is [89.6, 89.8] degrees across 144 attention modules. DPO also shows a clear linear mode-connectivity barrier and collapses late-layer CKA to approximately 0.46. Its effective rank is substantially higher (24.5 vs ~16 for the other methods), though with much smaller singular values given its 13x smaller Frobenius norm. DPO reaches 93.5% on GSM8K (McNemar p < 1e-9 vs each other method) and 30.0% on AIME26 against 3.3-10.0% for the rest, though again the learning rate confound applies.
Implications and Open Questions
The most practically significant finding is the near-collinearity of SFT, RFT, and RIFT. If reward filtering and linear reward weighting produce the same weight update as plain SFT, then the accuracy gains sometimes attributed to these formulations in the literature may be attributable to data curation, compute, or hyperparameter choices rather than the loss structure itself. This should prompt more careful ablations in future work claiming that a specific offline RL objective is responsible for improvements over SFT.
The DFT result is worth dwelling on. A stop-gradient on token probabilities, which has no effect on the gradient sign and only rescales individual token contributions, produces a more geometrically distinct update than methods that explicitly incorporate binary rewards. This suggests that the implicit curriculum imposed by confidence-based weighting has a stronger structural effect than reward filtering on this data distribution, at least under attention-only LoRA.
DPO's orthogonality to the SFT cluster is consistent with its fundamentally different loss structure: contrastive log-ratio objectives optimise a different quantity than NLL-based methods, and the geometry reflects this. The higher effective rank is intriguing and suggests DPO distributes its update across more singular directions, possibly because the contrastive objective requires simultaneously increasing and decreasing probabilities in a way that NLL losses do not.
The layer-depth pattern in Offline GRPO's orthogonal component (growing from ~55% in middle layers to 79-86% in late layers) aligns with the CKA divergence the authors observe in the same region. Late transformer layers are generally understood to carry more task-specific and output-formatting information, so it is plausible that the group-relative advantage signal in GRPO preferentially modifies these layers relative to the SFT direction.
Limitations
The authors are appropriately candid about the study's scope. The analysis covers a single domain (math), a single base model, and attention-only LoRA adapters. Whether the geometric relationships hold for full fine-tuning, different modalities, or different base models is unknown. The AIME26 evaluation uses only 30 examples with greedy decoding, so those numbers should be treated as directional rather than definitive.
The DPO learning rate confound is the most significant methodological limitation. The 10x smaller learning rate is standard practice in DPO codebases, but it means the update norm, effective rank, and accuracy figures for DPO are jointly determined by the loss function and the optimiser setting. A matched comparison could substantially change the picture, and the authors acknowledge this explicitly rather than glossing over it.
The broader research context here is a growing body of weight-space fine-tuning analysis (Arturi et al., 2025; Soligo et al., 2025; Zhong and Raghunathan, 2025; Ward et al., 2025) that treats weight deltas as first-class objects of study. This paper is a useful contribution to that programme specifically for the offline RL setting, and the release of code, adapters, and analysis scripts should make it straightforward for others to extend the comparison to additional methods or base models.
Full paper: arXiv:2606.23740