The query-key-value (QKV) formulation has been a fixed feature of transformer attention since Vaswani et al. (2017), yet surprisingly little work has asked whether all three projections are independently necessary. As context windows grow and on-device deployment becomes a practical target, the KV cache has emerged as a primary memory bottleneck during autoregressive generation. This paper, available at arXiv:2606.04032, takes a systematic approach to answering that question: what happens when you tie some or all of these projections together, and does it actually matter for downstream quality?
The motivation is clean. Self-attention projections account for roughly 30% of transformer parameters, and during generation the cached K and V tensors can dominate GPU memory for long-context workloads. If some of that redundancy can be removed without meaningful quality loss, the practical payoff is substantial: longer effective context windows, higher serving throughput, and lower infrastructure cost. The paper frames projection sharing as an underexplored instance of weight tying, distinct from but composable with existing head-sharing methods like grouped-query attention (GQA) and multi-query attention (MQA).
What Was Tested
Three projection-sharing constraints are evaluated against a standard QKV baseline:
- Q=K-V: Query and Key share a projection; Value is separate. This produces symmetric attention maps.
- Q-K=V: Query is separate; Key and Value share a projection. Attention patterns remain asymmetric.
- Q=K=V: A single projection serves all three roles.
Because Q=K and Q=K=V both produce symmetric attention matrices, the authors also test augmented variants using 2D positional encodings to reintroduce asymmetry, denoted with a superscript +. Experiments span synthetic sequence tasks, vision benchmarks (MNIST, CIFAR, TinyImageNet, anomaly detection), and language modelling at 300M and 1.2B parameter scales trained on 10B tokens from SlimPajama. All models are trained from scratch with matched hyperparameters, which is the right approach for isolating architectural effects.
Key Results
The headline finding is that Q-K=V is the only projection-sharing variant worth deploying in language models. At 300M parameters it achieves 3.1% perplexity degradation relative to the QKV baseline while halving KV cache memory. At 1.2B parameters the degradation narrows to 2.48%, suggesting the constraint becomes easier to satisfy at scale. On five-shot downstream benchmarks (HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande), the average accuracy drop is only 0.41 percentage points, a gap that is arguably negligible for most production use cases.
The asymmetry between the two two-projection variants is the most instructive result. Q=K-V and Q-K=V have identical parameter counts and identical computational cost, yet Q-K=V achieves better validation perplexity (5.27 vs 5.36 at 300M scale) and provides 50% cache reduction while Q=K-V provides none. This is not a trivial finding. It directly implies that the Value projection is less critical than the Key projection for maintaining model quality, and that forcing Q=K breaks the directional asymmetry that causal language modelling requires. The authors support this empirically by measuring cosine similarity between projection matrices in trained QKV models: K and V exhibit high similarity (0.73 on average) and comparable effective rank (687 vs 702 out of 1024 dimensions), while Q maintains lower similarity with both K (0.42) and V (0.31). This is a plausible mechanistic account, even if it remains empirical rather than formal.
Q=K=V performs poorly, with 25.4% perplexity degradation. This is not surprising given that it combines symmetric attention with the loss of independent value routing, but it is useful to have the number confirmed at scale.
The paper also evaluates combined projection-plus-head sharing. Q-GQA-4 achieves 87.5% cache reduction with 3.9% perplexity degradation; Q-MQA reaches 96.9% cache reduction with 4.8% degradation. These combined variants occupy a region of the efficiency-quality Pareto frontier that neither approach reaches alone, which is the practical point: projection sharing and head sharing address orthogonal axes of redundancy.
Methodological Strengths and Weaknesses
The experimental design is careful. Training from scratch with matched hyperparameters avoids the confound of fine-tuning dynamics. Testing across synthetic, vision, and language domains gives a reasonable picture of where each variant succeeds or fails. The observation that Q=K attention works well for non-sequential tasks like image classification but fails for causal language modelling is well-supported and makes intuitive sense: image patches have no inherent temporal ordering, so symmetric attention is less of a liability.
The KV cache analysis is concrete and practically grounded. The cost projections for a 32k-context serving scenario ($6k/month savings with Q-K=V) are illustrative rather than rigorous, but the underlying memory arithmetic is straightforward and correct.
Several limitations are worth flagging. The largest scale tested is 1.2B parameters, and the authors explicitly note that behaviour beyond 7B is unconfirmed. Given that the degradation trend improves with scale, it is plausible that Q-K=V becomes even more attractive at larger sizes, but this is speculative. Evaluation is also capped at 2048-token sequences, which is short relative to the long-context scenarios the paper motivates. The theoretical account of why K=V works is empirical rather than derived; a more formal analysis of the low-rank structure of attention would strengthen the argument. The Q=V ablation is omitted, which the authors justify on the grounds that Q is not cached during generation, though it would have been useful for completeness.
Implications for Deployment
The practical case for Q-K=V is straightforward. A 50% reduction in KV cache memory translates directly to either doubling the effective context length for a fixed memory budget or doubling serving throughput by fitting more concurrent users per GPU. Combined with MQA, the cache reduction reaches 96.9%, which is close to the theoretical minimum. These gains require no changes to training infrastructure, no specialised initialisation, and no post-training compression steps. The models simply converge without instability.
The broader implication is that the standard QKV formulation carries redundancy that has gone largely unexamined. Weight tying between K and V is not a novel idea in principle, but this paper provides the first systematic, multi-domain, multi-scale characterisation of the trade-offs involved. The result that Q=K-V is worse than Q-K=V despite having the same parameter count is particularly useful for practitioners: the choice of which projections to tie matters more than how many you tie.
For researchers working on efficient transformers, this work sits naturally alongside GQA, MQA, and linear attention methods. The finding that projection sharing and head sharing are complementary rather than competing suggests that production models could reasonably combine both without choosing between them. Whether this holds at frontier scales (70B+) remains an open question, but the evidence at 1.2B is encouraging enough to warrant investigation.
The code is publicly available at github.com/anushamadan02/Do-Transformers-Need-3-Projections, and the full paper can be read at arXiv:2606.04032.