The performance gap between reinforcement learning-trained reasoning models and their supervised fine-tuned counterparts on mathematical benchmarks is well established. What has remained unclear is why this gap exists at a mechanistic level. This paper, available at arXiv:2607.26119, takes a step toward answering that question by examining internal representational structure rather than just output behaviour. The approach is methodologically grounded and the findings, while preliminary, offer genuine insight into how training objectives reshape computational architecture in transformer models.
Key Contributions
The paper presents three converging lines of analysis across two model families: DeepSeek-Math-7B (Instruct and RL variants) and OLMo-3 (Instruct and Think variants). The contributions break down as follows:
- Linear probing of layer-wise hidden states to measure how cleanly correct versus incorrect solutions are encoded at each transformer layer.
- Mean ablation studies that replace individual layer activations with their mean values, quantifying each layer's functional importance to reasoning accuracy.
- Token coefficient of variation analysis across 50 repeated samples per problem, treating generation consistency as a proxy for policy stability and adaptive compute allocation.
The use of synthetic problem generation (1,000 problems across four fixed templates covering probability, fractions, and cost calculations) is a sensible design choice. It eliminates data contamination concerns and allows algorithmic verification of ground truth, addressing a persistent confound in evaluations that rely on human-authored benchmarks where memorisation effects are difficult to rule out.
Methodology and What It Reveals
The linear probing setup is straightforward: binary classifiers trained on hidden states at each layer to predict whether a model's solution is correct. Probe accuracy serves as a proxy for representational clarity, the degree to which correctness information is linearly separable in activation space. This is a well-established technique in the interpretability literature, and its application here is appropriate given the research question.
The ablation metric, Accuracy Drop (AD), measures how much model accuracy degrades when a given layer's activations are replaced by their mean. Pearson correlation between layer depth and AD then captures whether importance concentrates in deeper layers (positive r) or is distributed uniformly (r near zero).
The results are fairly clear. RL-trained models achieve probe accuracies of 83-98% versus 75-90% for instruction-tuned models, with correctness representations emerging at or near layer 0 for RL models compared to layer 6 for DeepSeek-Math-Instruct. The ablation analysis shows RL models have a positive depth-importance correlation (r = 0.47), meaning deeper layers carry disproportionate weight for reasoning, whereas instruction-tuned models show near-zero correlation (r = -0.11), indicating uniform distribution of functional importance across the network. This architectural difference is the most substantive mechanistic finding in the paper.
The token variability analysis is more nuanced and, arguably, the most thought-provoking section. Coefficient of variation (CV) across 50 samples per problem reveals that DeepSeek-Math-RL exhibits substantially higher variability than DeepSeek-Math-Instruct, peaking around CV = 1.3 in the mid-difficulty range. OLMo-3-Think and OLMo-3-Instruct, by contrast, both maintain low, stable CV values (0.1-0.125) regardless of difficulty. The authors interpret high variability as evidence of an under-determined policy, where the model has not converged on a stable reasoning strategy. This framing connects productively to information-theoretic work on token complexity, though the causal story remains speculative at this stage.
Results and What They Mean
The central finding is that RL training reshapes how models internally encode reasoning. The earlier emergence and higher linear separability of correctness representations in RL models suggests these models are not simply generating longer chains of thought; they are structuring information differently from early layers onward. The concentration of functional importance in deeper layers, rather than uniform distribution, implies a form of hierarchical processing that may be more efficient for multi-step reasoning tasks.
The token variability results complicate any simple narrative about RL being uniformly beneficial. The OLMo-3 family shows that consistent, low-variability generation is achievable with both SFT and RL training, while DeepSeek-Math-RL shows the opposite pattern relative to its SFT counterpart. This strongly suggests that the specific training pipeline, reward structure, and base model interact in ways that cannot be reduced to a clean RL-versus-SFT dichotomy. The authors are appropriately cautious about overgeneralising here.
One observation worth flagging: the emergence layer analysis requires a dynamically defined threshold because some models exceed 80% probe accuracy at layer 0. This is methodologically reasonable but means the emergence layer metric is not strictly comparable across all model pairs. The authors acknowledge this, though it limits the interpretive weight one can place on specific emergence layer numbers.
Limitations and Open Questions
The sample sizes are modest. Ablation experiments use 20 GSM8K problems per model, and the token variability analysis uses 50 problems from GSM8K-Platinum. These are sufficient for preliminary findings but insufficient for strong statistical claims, particularly given the variance in model behaviour across problem types. The per-template probe accuracy figures in the appendix suggest some template-specific variation that the main results aggregate over.
The analysis is also restricted to 7B-parameter models. Whether the architectural differences observed here, particularly the depth-importance concentration in RL models, scale to larger models is an open question. There is some reason to expect the pattern to hold, but it is not guaranteed, and the field has seen enough scale-dependent surprises to treat this as a genuine uncertainty.
The paper does not examine intermediate reasoning steps, only final answer correctness. Probing for how multi-step solutions are constructed and validated internally would be a natural and valuable extension. Similarly, extending this analysis to domains beyond mathematics, such as code generation or scientific reasoning, would test the generality of the mechanistic claims.
The token variability interpretation as a measure of policy stability is conceptually appealing but would benefit from more formal grounding. The connection to Lee et al.'s token complexity framework is suggestive rather than conclusive, and the authors are right to present it as partial empirical support rather than confirmation.
Despite these limitations, the paper makes a meaningful contribution by moving the conversation about RL reasoning models from benchmark performance toward mechanistic explanation. The convergent methodology, combining probing, ablation, and generation analysis, is well chosen, and the finding that training objectives reshape internal computational architecture rather than merely output behaviour is worth taking seriously. Full details and code are available at https://arxiv.org/abs/2607.26119.