Post-training quantization (PTQ) has become a practical necessity for deploying large language models under memory and compute constraints. The dominant approach, rooted in GPTQ and the Optimal Brain Surgeon framework, solves a layer-wise weight update analytically using a frozen second-order approximation. This works well enough at moderate compression ratios, but it rests on approximations that compound quietly as quantization proceeds column by column through each layer. A new paper, REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent, identifies the structural source of this degradation and proposes a concrete fix that produces the best KL divergence results reported to date across a broad sweep of LLaMA-3.1 and Qwen3 models.
The Problem with Static Hessian Solvers
To understand why REAL-Q matters, it helps to be precise about what existing methods actually optimise. GPTQ minimises a layer-local MSE between the outputs of the full-precision and quantized weight matrices, given full-precision input activations. This introduces two misalignments simultaneously. First, by the time a given layer is being quantized, the activations it receives in practice will already be perturbed by upstream quantization errors; using clean activations during calibration ignores this entirely. Second, the layer-local MSE weights all output channels equally, whereas the true objective, KL divergence between the original and quantized model outputs, depends on a non-linear, highly coupled path through all subsequent transformer blocks and the language modelling head.
Recent work has addressed each blind spot in isolation. GPTAQ recalibrates input activations to correct upstream errors but retains a uniform reconstruction MSE. GuidedQuant incorporates end-to-end gradients to capture downstream sensitivity but still groups output rows, drops cross-channel coupling, and freezes the resulting Hessian for the entire layer. The authors frame this frozen-Hessian behaviour as information misalignment: the loss surface shifts as columns are progressively quantized, but the second-order information used to guide compensation is never updated to reflect that shift. Because maintaining analytical tractability requires these approximations, all methods in this lineage face the same structural constraint.
Key Contributions
REAL-Q breaks from the closed-form solver paradigm entirely. Its three main components are:
- Aggregated Fisher MSE: Rather than a uniform or saliency-weighted reconstruction loss, REAL-Q uses a second-order Taylor expansion of the global KL divergence at the current transformer block's output. The full aggregated Fisher matrix is retained, preserving cross-channel coupling that grouped baselines must discard. Crucially, the expansion point is chosen so that backpropagation stays confined to at most two adjacent blocks, keeping the computational overhead bounded.
- Dynamic Block-wise Gradient Descent (Block-GD): After every 128-column block within the GPTQ column sweep, REAL-Q applies a single Adam step using a gradient recomputed against the actual, partially quantized weights at that moment. This raises correction granularity from per-layer to per-column-block, roughly two orders of magnitude finer, and allows the optimiser to respond to the current state of the loss surface rather than an approximation computed before quantization began. Adam's diagonal preconditioner provides implicit per-coordinate scaling without requiring a Hessian inversion.
- Loss Sliding Window: Because the Fisher MSE objective is defined at the transformer block level, transitions between blocks could introduce discontinuous jumps in the loss. A sliding window mechanism interpolates the objective across block boundaries, stabilising the sequential optimisation process.
The paper also provides theoretical grounding. Under a linear structural-drift assumption, static solvers accumulate errors at O(n) rate as columns are quantized. For the dynamic first-order updates, a descent lemma argument shows that the true global loss decreases after a surrogate gradient step provided the cosine similarity between the surrogate gradient and the true KL gradient exceeds a threshold proportional to the learning rate and the smoothness constant of the objective. The authors verify empirically that REAL-Q's Fisher MSE gradient maintains cosine similarity well above this threshold, whereas the saliency-weighted MSE used by GuidedQuant and the uniform MSE used by GPTQ/GPTAQ fall substantially below it for many modules.
Results
Experiments cover LLaMA-3.1-8B and 70B, and Qwen3 at five scales from 0.6B to 32B, all at W4A16 with per-row weight quantization and QuaRot rotation preprocessing applied uniformly across all methods. KL divergence on WikiText-2 is the primary metric, supplemented by perplexity and zero-shot accuracy on ten downstream tasks.
REAL-Q achieves the lowest KL on every model evaluated. The headline reduction is approximately 49% relative to GuidedQuant on Qwen3-1.7B (KL 6.07 versus 11.9). On LLaMA-3.1-8B the improvement is more modest but consistent (KL 3.36 versus 3.67). On LLaMA-3.1-70B, where calibration is limited to 256 samples for tractability, REAL-Q still leads (KL 14.3 versus 15.1). Perplexity improvements track the KL results closely. Downstream accuracy gains are smaller and less uniform, which is expected given that zero-shot task accuracy is a coarser signal than distributional fidelity.
One noteworthy implementation detail: on smaller Qwen3 models (0.6B, 1.7B, 4B), the authors enable output-perturbation clipping at the 95th percentile when computing the Fisher MSE. These post-trained models exhibit a small number of channels with anomalously large output perturbations that can dominate the Fisher loss and misdirect the Adam updates. This is a sensible engineering fix, but it does flag that the method's behaviour is somewhat architecture-sensitive at small scale.
Limitations and Open Questions
The authors are candid about the costs. Block-GD introduces additional compute and memory overhead during calibration relative to a single-pass analytical solver. For 70B models this requires an FSDP multi-GPU pipeline to cache Fisher matrices and saliency information to disk before the per-layer quantization pass. The paper defers detailed profiling to appendices, and notes that system-level optimisations such as memory-efficient backward kernels and activation offloading could reduce this overhead in future work.
The theoretical descent guarantee applies to SGD; the authors acknowledge it does not transfer directly to Adam because the preconditioned step direction is no longer aligned with the surrogate gradient. The guarantee is therefore empirical for the actual optimiser used, which is a gap worth closing formally.
Gradient alignment degrades on FFN modules, particularly up_proj and down_proj, relative to attention modules. The SwiGLU non-linearity makes the block-wise truncated second-order expansion a poorer local approximation of the true downstream loss in these layers. This is an honest limitation: the Fisher MSE is a better surrogate for some architectural components than others, and the gains on models where FFN modules dominate may be harder to extract.
The evaluation is restricted to W4A16. Results at W3A16, W2A16, and weight-activation settings (W4A4KV4) are mentioned but not the focus of the main analysis, and it would be useful to see whether the gains are as consistent at more aggressive compression ratios where quantization error is larger and the benefits of tighter loss alignment might compound more strongly.
REAL-Q represents a meaningful step away from the closed-form solver paradigm that has dominated PTQ since GPTQ. The core insight, that freezing second-order information across an entire layer is a structural bottleneck rather than an incidental approximation, is well-motivated and the empirical results support it clearly. Whether the calibration overhead can be reduced to the point where REAL-Q becomes a drop-in replacement for GPTQ in standard inference pipelines is the practical question that will determine its adoption.
The full paper, including pseudocode, proofs, and extended experimental tables, is available at arxiv.org/abs/2609.00049.