As language models are increasingly deployed for long-context reasoning and agentic tasks, memory efficiency at inference time has become a serious engineering constraint. Softmax attention's KV cache grows linearly with sequence length, and while hybrid architectures using Gated DeltaNet (GDN) or Kimi Delta Attention (KDA) replace most attention layers with fixed-size recurrent states, those states are typically stored in FP32. At batch size 256 on Qwen3.6-35B, recurrent states alone consume 15 GB of GPU memory and account for over 24% of decoding latency. This paper, DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization, is the first systematic study of post-training quantization applied specifically to these recurrent states, and it identifies both why naive quantization fails and how to fix it.
The Problem with Uniform Quantization
The authors begin with a straightforward empirical question: can you just quantize the recurrent state to INT8 or FP8 and call it a day? The answer is a clear no. On complex mathematical reasoning benchmarks like AIME 2026, INT8 quantization drops accuracy from 85.46% to 18.48% on Qwen3.6-35B. FP8 is similarly catastrophic. INT4 and NVFP4 effectively destroy mathematical reasoning and code generation entirely, reducing scores to near zero.
This is qualitatively different from weight quantization, where 8-bit formats typically impose modest accuracy penalties. The reason is architectural. In GDN and KDA, the recurrent state is not written once and read many times; it is read, transformed by a learned decay gate, updated via a delta-rule correction, and written back at every decoding step. A quantization error introduced at step t is not discarded; it propagates through every subsequent recurrence, amplified or attenuated by the decay dynamics. The paper formalises this error-feedback process and shows that accumulated quantization error is shaped by both the magnitude of the initial residual and the persistence of that channel under the decay gate.
Key Contributions and Methodology
DAMP is built on two empirical observations about the structure of GDN and KDA recurrent states:
- Error concentration: Quantization error is not uniformly distributed across channels. Even after applying the Hadamard transform (which is standard practice for smoothing outliers), INT8 reconstruction error remains concentrated in a small subset of key channels. This heterogeneity is the fundamental reason uniform precision fails.
- Decay stability: While instantaneous decay values vary token by token, the ordering of channels by decay strength is stable across prompts and tasks. Channels that decay slowly persist longer in the state, meaning quantization errors in those channels accumulate over more steps.
DAMP combines these two signals into a per-channel risk score during offline calibration. Channels with high quantization-error energy and slow decay (high persistence) receive the highest risk scores and are stored at higher precision. The remainder are stored in INT8. The resulting mixed-precision layout is static: it is computed once and reused throughout inference, with no token-wise or prompt-wise adaptation required. This is a practically important property, since dynamic precision selection would introduce significant runtime overhead.
The method requires no retraining and no modification to model weights. It is purely a post-training compression strategy applied to the state tensors at serving time.
Results
The experiments cover two large open-weight hybrid MoE models: Qwen3.6-35B-A3B and Kimi-Linear-48B-A3B-Instruct, evaluated across six benchmarks spanning mathematical reasoning (AIME 2026, HMMT Feb, IMO-Ans), general reasoning (GPQA-Diamond, MMLU-Pro), and code generation (LCB-v6).
At 9.9 effective bits per state value, DAMP achieves the following relative to FP32 baselines:
- On Qwen3.6-35B, AIME 2026 accuracy drops from 85.46% to 83.65%, compared to 44.21% for INT8+Hadamard at 9.0 bits.
- On Kimi-Linear-48B, AIME 2026 accuracy is 63.72% versus 62.34% for FP32 and 55.72% for INT8+Hadamard.
- Recurrent-state storage is reduced by 69.1%.
- The recurrent-state update kernel accelerates by up to 2.01x.
- Full-model time per output token (TPOT) decreases by up to 10.9%.
A notable finding from the baseline comparisons is that BF16 consistently underperforms FP16 despite identical storage cost. This suggests recurrent states benefit more from fine quantization resolution (mantissa bits) than from extended exponent range, which has implications for format selection in future serving systems. The INT8+Hadamard baseline outperforms FP8 on both models, reinforcing this point.
Limitations and Open Questions
The paper is technically careful and the results are strong, but several questions remain worth considering. First, the evaluation is limited to two model families. Both Qwen3.6 and Kimi-Linear are hybrid MoE architectures with a specific ratio of recurrent to attention layers. It is not immediately clear how DAMP's calibration procedure would transfer to models with different decay distributions or different ratios of GDN/KDA layers, such as purely recurrent architectures.
Second, the static layout assumption deserves scrutiny. The authors show that decay ordering is stable across prompts and tasks in their evaluation set, but this stability may not hold for highly domain-shifted inputs or for models fine-tuned on specialised data after the calibration set was constructed. The sensitivity of the layout to calibration set choice is not extensively analysed.
Third, the 9.9 bits effective cost is not dramatically lower than INT8 at 9.0 bits. The gain over INT8+Hadamard is substantial in accuracy terms, but the storage improvement relative to a well-tuned INT8 baseline is modest. For applications where the primary constraint is storage rather than accuracy, the trade-off profile may look less compelling than the headline 69.1% reduction over FP32 suggests.
Finally, the paper focuses on decoding latency. Prefill-phase behaviour, where recurrent states are built up from long input contexts, is not characterised in detail. For long-document workloads, prefill costs can dominate, and the interaction between quantization error accumulation and long prefill sequences may differ from the decoding regime studied here.
These are relatively minor gaps in what is otherwise a well-motivated and well-executed paper. The core insight, that recurrent-state quantization requires joint consideration of error magnitude and decay-based persistence, is both novel and practically actionable. As hybrid architectures with linear-attention layers become more common in production deployments, methods like DAMP will be directly relevant to serving infrastructure design. The paper is available at arXiv:2608.27513.