Three of the most widely used methods for training language models to reason, GRPO, Dr. GRPO, and DAPO, are typically presented as distinct algorithmic choices with separate motivations. A new paper on arXiv (arXiv:2607.00152) argues they are not. All three are operations on a single quantity: the standard deviation of rewards within a sampled group of answers to one prompt. The paper derives an exact, finite-group identity that makes this concrete, and the result has immediate practical implications for how researchers should think about group size, difficulty bias, and compute allocation in RLVR training.
The Problem Being Solved
Reinforcement learning with verifiable rewards (RLVR) works by sampling multiple answers to the same prompt, having an external verifier mark each one correct or incorrect, and then updating the model to favour correct paths. The key object in this loop is the advantage: a number assigned to each response that tells the optimizer how much better or worse it was than the group average.
GRPO computes advantages by subtracting the group mean reward and dividing by the group standard deviation. Dr. GRPO drops that division. DAPO discards groups where all answers agree, which corresponds exactly to groups where the standard deviation is zero. These choices have been discussed as separate design decisions with separate justifications, but the paper shows they share a common algebraic structure that has not been made explicit before.
The core observation is simple but consequential. For binary rewards, the group reward standard deviation is σ = √(k(G−k))/G, where G is the group size and k is the number of correct answers. This quantity is zero when the group is unanimous and maximised when answers are split evenly. It is, literally, the amount of within-prompt disagreement. The paper proves that one GRPO gradient step on a single prompt equals σ times the contrast between the scores of correct and incorrect responses. The standard deviation is not just a normaliser; it is the magnitude of the learning signal itself.
Key Contributions
- The group-standard-deviation identity: A finite-group, baseline-free derivation showing that the GRPO update for a single prompt factors as σ times a contrast term. This holds for any group size and any policy dimension.
- Unification of three methods: GRPO scales by 1/σ (normalising the signal), Dr. GRPO omits that scaling (preserving the raw signal magnitude), and DAPO filters out groups where σ=0. Three methods, three operations on one number.
- Objective-level interpretation: Dividing by σ causes GRPO to ascend the arcsine-transformed success rate 𝔼[2 arcsin √p], the classical variance-stabilising transform of a binomial proportion, rather than the raw success rate. Dr. GRPO ascends the raw success rate directly. This connects to prior work by Thrampoulidis et al. on surrogate reward views of GRPO.
- Closed-form group-size law: The paper derives G ≳ 1/(8εp(1−p)) as the group size needed to realise a target fraction (1−ε) of the large-group gradient. This makes explicit that the required group size scales inversely with difficulty-adjusted variance, and that easy and hard prompts are systematically underserved by a uniform G.
- Silent-group rate: The fraction of groups providing no usable signal is p^G + (1−p)^G. Both the group-size law and the silent-group rate are validated on a corpus of 215,608 problems with measured per-prompt difficulty.
Why the Standard Deviation Division Matters
The paper's treatment of the σ division is the most analytically valuable part. Liu et al. (Dr. GRPO) identified this step as the source of a difficulty bias and removed it. The present paper explains precisely what that bias is: dividing by σ re-weights prompts so that those at intermediate difficulty (p ≈ 0.5) receive less marginal weight relative to very easy or very hard prompts, because σ is largest in the middle. The arcsine transform flattens this out. Whether this re-weighting is desirable depends on the training goal.
If the goal is to maximise raw accuracy, Dr. GRPO's objective is better aligned. If the goal is to apply sustained pressure to hard prompts that would otherwise receive small updates because their success rate is low, GRPO's standardisation helps. Neither is universally correct. The paper's contribution is to make this trade-off explicit rather than leaving it implicit in the choice of normalisation.
DAPO's dynamic sampling, which discards all-correct and all-incorrect groups, is shown to be exactly equivalent to conditioning on σ > 0. The discarded gradient mass is a closed-form function of the per-prompt solve rate and the group size, which means its effect on training dynamics is predictable rather than empirical.
Limitations and Open Questions
The authors are careful about scope. The identity is exact for binary rewards and on-policy sampling. It deliberately sets aside reward clipping, KL penalties, off-policy staleness, and non-binary reward signals. The controlled validation uses a Bernoulli-logit policy where the score is scalar; a full language model training run that logs silent-group rates and per-difficulty gradient mass across varying G would be the natural next test.
The DAPO comparison rests on a structural identification of discarded gradient mass, with a time-evolution fit serving as a consistency check rather than an independent prediction. A logged per-prompt solve-rate history would allow the all-correct curve to be predicted rather than retrospectively fit, which would strengthen the empirical case.
There is also a downstream question the paper does not address: whether the extra gradient weight that standardisation places on hard prompts actually improves generalisation beyond the trained difficulty range. This connects to the broader question of when models extrapolate past their training distribution, which remains open.
The framework the paper establishes, starting with one prompt, one sampled group, and one exact update, is presented as a template for analysing other reward-shaping choices. Rank-based advantages, quantile advantages, reward clipping, and length normalisation are all candidates for the same single-group accounting. That programme is left for future work, but the identity derived here makes the approach credible.
For researchers working on RLVR training pipelines, the practical upshot is that group size is not a free hyperparameter. It is a difficulty-dependent compute budget with a known efficiency curve. Uniform group sizes are a reasonable default, but the group-size law tells you exactly where that default fails, and the silent-group rate tells you how much compute is being spent on uninformative rollouts. Both are now computable from a difficulty distribution rather than requiring ablation studies.
The full paper is available at https://arxiv.org/abs/2607.00152.