One of the more persistent puzzles in transformer training is understanding what properties of training data actually determine how much a model's weights change. Loss curves and downstream benchmarks provide indirect signals, but they offer little mechanistic insight into the weight-level dynamics. This paper (arXiv:2608.23573) takes a different approach: it asks whether a single, training-free statistic computed from raw corpus data can predict how much the Weibull scale parameter of a transformer's weight distribution grows during training. The answer, within a controlled setting, is yes, with a law that collapses 23 runs spanning an order of magnitude in learning rate onto a single curve.
The Core Idea
The starting point is an empirical regularity from prior work: the magnitudes of weights in transformer linear layers follow a two-parameter Weibull distribution, and the shape parameter k sits remarkably close to 1.2 across layers, models, and training stages. Because k barely moves, the scale parameter λ carries essentially all the training-induced signal. The question then becomes: what property of the training corpus determines how much λ grows?
The paper's answer is the bigram conditional entropy D = H(next | prev), computed directly from token co-occurrence counts before any training begins. This measures how much the previous token constrains the next: low D means the local next-token mapping is tight and predictable; high D means it is diffuse. The proposed law is:
λ² − λ₀² = C₀(η) + C₁(η) · (H_r − D)^0.59
where λ₀ is the initial scale, H_r is the saturation value of the plug-in bigram estimator under matched-budget shuffling (approximately 8.0 bits at the 2.4M-token budget used), and η is the learning rate. The two coefficients C₀(η) and C₁(η) absorb the learning rate dependence and follow their own power-law scalings: C₀ ∝ η^1.55 and C₁ ∝ η^1.75, both with R² above 0.99 across four learning rates.
What Makes This More Than a Curve Fit
The authors are careful to distinguish their result from an unconstrained regression. Two features are worth highlighting.
First, the exponent 0.59 is not selected by fitting the weight-scale growth curve directly. It is derived from a separately measured data-side saturation relation, where the plug-in bigram entropy saturates as a power law in the retained-structure fraction S. The exponent 0.59 = 1/p with p ≈ 1.69 is carried over from that independent measurement. This means the functional form is constrained by a separate empirical relation, not introduced as a free parameter to improve the growth fit.
Second, after removing the two per-learning-rate coefficients, 23 runs collapse onto the single curve y = (H_r − D)^0.59 with slope 1.00 and R² = 0.941. The authors are transparent that per-learning-rate fits are weaker (R² ≈ 0.82), and they report both numbers explicitly rather than presenting only the more flattering collapsed result. The collapse tests the shared exponent and unit slope; the per-η accuracy is the practically relevant figure for forward prediction.
Methodology and Validation
The experimental design is clean. The authors hold the corpus fixed (wikitext) and vary only the corruption level ρ, which shuffles a fraction of within-sequence token positions. This isolates the local next-token mapping as the single varying quantity. The bigram entropy D and the structure-retain fraction S are computed from the raw token counts before training; no model outputs are used on the data side.
Key methodological choices include:
- Weibull fitting is applied to transmission-class matrices only (attention output projection, feed-forward up and down projections); query/key/value projections are excluded because their magnitude distributions follow a different regime.
- The pooled model-level λ² is well approximated by the parameter-count-weighted average of per-block λ² values, because second moments are additive and k varies only weakly across blocks (1.16 to 1.21).
- The main analysis uses the non-saturated corruption range ρ ≤ 0.6, where the plug-in bigram estimator retains resolution.
- An end-to-end self-validation predicts held-out corruption levels from the data statistic alone, recovering measured λ with 5.7% relative error.
The law is also tested across two architectures: Pythia-70m (GELU feed-forward, LayerNorm) and a Llama-style 70M variant (SwiGLU, RMSNorm). Both follow the same convex form with the same exponent; only the coefficients differ. The Llama-style model shows roughly twice the data sensitivity in its feed-forward blocks (C₁ ≈ 0.74 to 0.85 versus 0.32 to 0.41 for Pythia's GELU layers), which the authors attribute to the gated nonlinearity converting predictable structure into weight-scale growth more efficiently. The Weibull shape parameters are essentially matched across architectures (k = 1.204 versus 1.207), confirming that the coefficient difference reflects a genuine weight-scale response rather than a distributional shape artifact.
Limitations and Open Questions
The paper is notably candid about where the law breaks down. Several limitations deserve attention from anyone considering applying this framework.
The most significant boundary is the cross-corpus failure. Code, with D ≈ 5.30, falls well below the within-corpus law and is over-predicted by roughly six times the in-corpus error. The authors argue this implicates a second data dimension, redundancy, because low D can arise from either rich local mappings (natural text) or cheap repetition (code templates, tiled sequences). A single conditional-entropy variable cannot distinguish these cases. The proposed extension to a two-dimensional (D, R) framework is deferred to future work, and the evidence for redundancy as the missing axis currently rests on a single off-law corpus.
Other scope constraints are worth noting:
- The law is established at 70M parameters and approximately 10^8 training tokens, well inside the transient regime before weight-decay steady state. Whether the convex form and its coefficients carry to larger models or longer training is untested.
- H_r = 8.0 is a plug-in estimator ceiling set by the token budget, not the information-theoretic marginal entropy (10.7 bits). It should be read as a matched-budget baseline, not an intrinsic corpus property.
- Some corruption levels and learning rates are single-seed, so coefficient magnitudes at those points are indicative rather than precisely estimated.
- The architecture comparison conflates the feed-forward gating with normalization differences (RMSNorm versus LayerNorm), so isolating the contribution of gating specifically requires a controlled ablation not performed here.
The broader implication is that the Weibull weight-scale statistic offers a weight-trajectory-level readout of data structure that is complementary to existing data-quality methods. Most current approaches rely on downstream task performance or perplexity-based proxies evaluated after training. A pre-training predictor of weight-scale growth, even one currently limited to within-corpus corruption gradients, points toward a different kind of data characterisation: one grounded in how the corpus structure is written into the weight distribution during training. Extending this to a genuine cross-corpus, multi-dimensional framework, and testing it at scale, is the natural next step.
The full paper is available at https://arxiv.org/abs/2608.23573.