One of the more persistent frustrations with large reasoning models is that correctness-based training signals say almost nothing about how a model reasons, only whether it gets the right answer. A model that wanders through five pages of redundant scratch work before landing on a correct solution is treated identically to one that reasons efficiently and directly. ERR+ addresses this gap by introducing a reward signal grounded in the internal entropy dynamics of the thinking phase, offering a principled way to encourage decisive, well-structured reasoning without sacrificing exploratory capacity.
The Core Observation
The paper begins with an empirical finding that is simple but consequential: correct reasoning traces, across multiple model families and benchmarks, exhibit more frequent and larger token-level entropy drops than incorrect ones. Entropy here is the standard Shannon entropy computed over the model's output distribution at each token position during the thinking phase. High-entropy tokens correspond to genuine decision points where the model is uncertain about which direction to take. Low-entropy tokens reflect committed, near-deterministic generation.
What distinguishes correct traces is not lower average entropy overall, but a characteristic pattern of sharp drops following exploratory high-entropy phases. The model explores, then commits. Incorrect traces tend to remain at persistently elevated entropy, suggesting the model never quite resolves its uncertainty into a definite reasoning path. This is visualised clearly in the paper's Figure 1, which shows Gaussian-smoothed entropy trajectories for correct versus incorrect rollouts on AIME24, both at the single-case and dataset-aggregated level.
This framing is meaningfully different from prior entropy-based approaches. Methods like PEAR penalise high entropy directly as a proxy for verbosity, but this suppresses the exploratory positions that Wang et al. (2025) identify as the primary sites of RLVR learning gains. Penalising exploration to get shorter outputs is a blunt instrument that, predictably, degrades accuracy. ERR+ instead rewards the resolution of uncertainty, leaving high-entropy states unconstrained.
Methodology: Two Phases, One Sequential Pipeline
ERR+ operates in two sequential phases, each introducing a distinct reward component.
Phase 1: Entropy Relief Reward (ERR). The ERR bonus is proportional to cumulative token-level entropy drops within the thinking phase, log-normalised by response length to prevent the reward from simply scaling with trace length. Only drops in correct responses are rewarded, so the signal is conditioned on eventual accuracy. The log-normalisation is important: without it, longer responses would mechanically accumulate more drops and receive higher bonuses regardless of reasoning quality.
Phase 2: Robust Relative Efficiency Reward (RRER). Once Phase 1 has shaped the internal reasoning structure, Phase 2 introduces a length signal. Rather than penalising length in absolute terms, RRER scores each response's length against co-generated peers using a tanh-transformed within-group z-score. The tanh saturation prevents outlier responses from dominating the gradient signal, and the within-group normalisation makes the reward sensitive to prompt difficulty, since harder problems naturally elicit longer responses across the board.
The sequential design is not arbitrary. The paper provides a formal gradient-conflict analysis (Theorem 1) showing that in early training, the ERR and RRER objectives produce conflicting policy gradient directions. ERR pushes the model toward generating decisive entropy drops, which initially requires longer exploratory phases to set up those drops. RRER simultaneously penalises length. Applying both at once in early training creates a destructive interference that the sequential schedule avoids by waiting until Phase 1 has converged before introducing the length signal.
Experimental Results
The authors evaluate on four model backbones: DeepSeek-R1-Distill-Qwen-1.5B and 7B, and Qwen3-4B and 8B. Training uses 7,473 GSM8K problems, and evaluation covers five benchmarks: GSM8K, MATH-500, AIME24, AMC23, and MMLU-STEM.
Key results on DeepSeek-R1-Distill-Qwen-1.5B:
- ERR+ improves accuracy over GRPO while simultaneously reducing response length, a combination that no baseline achieves.
- GRPO improves accuracy but does not shorten responses. PEAR shortens responses but degrades accuracy. ERR+ does both.
- On AIME24, one of the harder out-of-distribution benchmarks, gains are meaningful given the small problem set (30 problems).
- The reward signal transfers to DAPO as the base optimiser, confirming it is not an artefact of GRPO-specific dynamics.
- The four-run mean on the primary evaluation is 66.10% ± 0.61% accuracy at 5794 ± 84 tokens, suggesting reasonable stability across seeds.
The anti-reward-hacking analysis in Section 6.1 is worth highlighting. A legitimate concern with any entropy-based reward is that the model might learn to manufacture artificial entropy cycles, inflating entropy then dropping it sharply without any genuine reasoning improvement. The authors check this by tracking the ratio of upward entropy movement to rewarded drops (Up/Drop) and the per-trace sign-change count. Both remain stable or decrease over training, which is the expected signature of genuine commitment events rather than artificial cycling. This is a necessary check and the paper handles it carefully.
Limitations and Broader Implications
The paper is candid about what it does not address. The reward signals optimise for accuracy and conciseness, but human evaluators often care about other properties: step-by-step clarity, pedagogical structure, or stylistic consistency. A model that reasons correctly and briefly is not necessarily one whose reasoning trace a student or practitioner can follow. Integrating preference-based or multi-objective signals alongside ERR+ would be a natural extension.
There is also the question of generalisation beyond mathematical reasoning. The entropy-drop phenomenon is characterised on maths benchmarks, where correct reasoning has a relatively clear structure. Whether the same dynamics hold for open-ended reasoning, code generation, or tasks without verifiable ground truth is an open question. The MMLU-STEM results provide some evidence of out-of-domain transfer, but this is a limited test.
The theoretical analysis, while useful for motivating the sequential design, rests on assumptions about the mid-exploration fraction and entropy surplus that are not directly measurable during training. The formal result is more of a motivating argument than a tight guarantee, which the authors acknowledge implicitly by treating Theorem 1 as justification for the design choice rather than a proof of optimality.
More broadly, ERR+ contributes to a growing body of work suggesting that the internal computational structure of reasoning traces, not just their outputs, carries actionable training signal. The connection to Wang et al. (2025)'s finding that the top-20% highest-entropy tokens account for the majority of RLVR gains is particularly interesting: ERR+ can be read as a method for ensuring those high-entropy positions are actually resolved rather than left dangling. Whether entropy dynamics generalise as a quality signal across modalities and task types seems like a productive direction for follow-up work.
The code is available at github.com/XrkArul/err_response and the full paper at arxiv.org/abs/2608.28771.