As large language models are deployed on increasingly long-context workloads, the KV-cache has emerged as a primary bottleneck. It grows linearly with input length, consuming substantial GPU memory and memory bandwidth during inference. A proliferation of compression techniques has followed, spanning quantization, token pruning, and token merging. The problem is that these methods have been evaluated in isolation, on different models, datasets, and serving configurations, making principled comparison nearly impossible. This paper, available at arXiv:2607.05399, addresses that gap directly with a workload-aware benchmark that jointly evaluates task quality and system performance across representative KV-cache optimisation strategies.
What the Paper Contributes
The core contribution is methodological: a unified experimental framework that holds models, datasets, prompt templates, and hardware constant while varying the compression mechanism. The authors evaluate four methods across three paradigms:
- KIVI (2-bit and 4-bit): asymmetric, data-dependent quantization that explicitly handles outlier structure in keys and values.
- TurboQuant (3-bit and 4-bit): a rotation-based, data-oblivious quantization approach with theoretical guarantees.
- SnapKV: attention-pattern-guided token pruning applied once after prefill, retaining salient tokens under a fixed cache budget.
- CaM: a merging-based approach that periodically redistributes evicted token contributions during decoding rather than simply discarding them.
Both Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 are evaluated across six LongBench tasks covering multi-document QA, single-document QA, few-shot learning, and summarisation. System metrics, including time-to-first-token (TTFT), output throughput, and prefill KV-cache memory footprint, are measured across context-length buckets using NarrativeQA, GovReport, and Qasper.
Key Findings
The headline result is that compression ratio is a poor predictor of end-to-end performance. This is not a trivial observation. It challenges a common intuition in systems work that a higher compression ratio straightforwardly translates to better throughput or lower latency. The results show this relationship is mediated heavily by workload type and model architecture.
- KIVI4 provides the most consistent task quality across both models and all task types, with minimal degradation relative to the full-cache baseline. KIVI2 is more aggressive and shows expected quality drops, particularly on Mistral.
- SnapKV delivers the strongest throughput gains under long-context conditions, benefiting from its post-prefill pruning strategy that reduces the token budget before the decode phase begins.
- CaM achieves striking quality gains on specific QA workloads, in some cases exceeding the full-cache baseline, but exhibits high variance across tasks. Its summarisation scores drop substantially, and its realised compression ratio varies widely depending on the dataset.
- TurboQuant is notably slow at prefill, with TTFT consistently higher than the full-cache baseline. On some configurations, it required three to four days per run for summarisation tasks, limiting its practical evaluation here.
- Quantization methods (KIVI, TurboQuant) tend to increase TTFT relative to baseline because of the overhead introduced during quantization and dequantization at prefill. Pruning and merging approaches, by contrast, reduce the effective token count and can lower TTFT at longer context lengths.
The finding that lossy compression can sometimes improve task accuracy is worth pausing on. SnapKV and CaM outperform the full-cache baseline on several QA benchmarks. The authors interpret this as an attention-focusing effect: by discarding less relevant tokens, these methods reduce noise in the attention computation and may sharpen retrieval. This is a genuinely counter-intuitive result from a data management perspective, where lossy compression is conventionally assumed to degrade output quality monotonically.
Methodological Considerations
The experimental setup is careful and reproducible, with code and data released publicly. However, several constraints are worth noting. All experiments run on a single NVIDIA A100 with 40GB of memory, which limits the evaluation to 7-8B parameter models. The authors acknowledge this and argue that the qualitative trends should generalise, since KV-cache scaling behaviour is fundamentally similar across model sizes. That argument is reasonable but not empirically verified here.
TurboQuant's GovReport results are omitted entirely due to runtime constraints, and summarisation results for TurboQuant are based on single runs. This is a real limitation for drawing conclusions about that method's behaviour on long-form generation tasks. The authors are transparent about this, but it means the comparison across methods is incomplete for the summarisation workload.
CaM's configuration involves several interacting hyperparameters, including compression interval, target cache size, hidden-state buffer size, and merge budget. The authors fix these to specific values without an ablation over the sensitivity of results to these choices. Given that CaM shows the highest workload sensitivity of any method tested, understanding how much of that variance is driven by configuration rather than the method itself would strengthen the conclusions.
The benchmark also does not evaluate multi-request batching scenarios, which are central to production serving. Single-request latency and throughput are useful, but the interaction between KV-cache compression and batching efficiency is a distinct and practically important dimension that remains unaddressed.
Implications for Deployment and Future Work
The practical guidance that emerges is clear: there is no universally superior KV-cache compression strategy. For deployments where quality stability across diverse workloads is the priority, KIVI4 is the safest choice. For long-context throughput-sensitive applications where the task type is known and relatively homogeneous, SnapKV offers meaningful gains. CaM's periodic merging strategy is promising for QA-style workloads but requires careful workload characterisation before deployment.
The broader implication for the research community is that future work on KV-cache optimisation should be evaluated against a shared benchmark rather than bespoke configurations. The authors position this paper as a starting point for that standardisation effort, and the framing is apt. The data management community's experience with query optimisation and workload-aware scheduling is directly applicable here, and the paper makes that connection explicit in its conclusions.
Open questions remain around how these methods interact with speculative decoding, continuous batching, and disaggregated prefill-decode architectures that are increasingly common in production systems. The relationship between compression strategy and attention mechanism variant (GQA vs MHA) also deserves systematic study, particularly as GQA becomes standard in newer models.
For anyone working on LLM inference infrastructure or evaluating compression strategies for long-context serving, this paper provides a useful empirical foundation. The full paper is available at https://arxiv.org/abs/2607.05399, with code at GitHub.