A quiet assumption has sat at the heart of transformer attention since the original Vaswani et al. (2017) formulation: that value vectors, like queries and keys, should be computed from the full residual stream. A new paper, Do Value Vectors in Deep Layers Need Context from the Residual Stream?, challenges this assumption directly, finding that in deeper layers the residual stream is not just unnecessary for value computation but may actively dilute the original token information that those layers actually need.
The practical upshot is a proposed architecture called Bank of Values (BoV), which replaces the standard dense value matrix in the last third of transformer layers with a static, per-token lookup table. The result is a model that improves validation loss and downstream benchmark scores at both 135M and 780M parameter scales, while consuming fewer FLOPs per token and requiring less KV cache memory than standard attention.
The Problem with Standard Value Computation
In a standard attention layer, the value vector is computed as x W_V, where x is the residual stream at that layer. The residual stream accumulates context from all previous layers, including cross-token information propagated through earlier attention heads. For queries and keys, this accumulated context is clearly beneficial: deeper layers need to know what other tokens have said in order to decide what to attend to. But value vectors are different. They do not participate in the attention score computation itself; they are simply what gets written back to the residual stream once attention weights are determined.
The paper's central empirical finding is that in deeper layers, the most useful component of the value vector is the part that reflects the original token identity, not the contextual information that has accumulated in the residual stream. Prior work such as Value Residual Learning (Zhou et al., 2024) had already shown that interpolating deeper-layer value vectors with the first layer's value vector improves performance. But those methods always added the context-free component on top of the standard context-dependent computation, leaving the question open: what if you dropped the context-dependent part entirely?
Methodology and the BoV Design
The authors run systematic ablations across four axes: which input vector to use for value computation, which layers to target, whether to include a learnable scaling coefficient, and whether to add or replace the standard value vector. The key finding from these ablations is that the gains attributed to prior methods come almost entirely from the context-free component. The context-dependent component contributes little to aggregate benchmark performance in deeper layers.
This observation has a convenient structural consequence. If the value vector for a given token in a deep layer depends only on the token's identity and fixed model parameters, then it is fully determined before any input sequence is seen. It can be precomputed and stored as a model parameter rather than computed at inference time. BoV formalises this: for each of the last third of layers, a value table E_v of shape |V| x d is learned, where |V| is the vocabulary size and d the hidden dimension. During a forward pass, the value vector at each position is simply a lookup into this table, scaled by a learnable per-layer scalar gamma_v.
The memory and compute implications are meaningful. These layers no longer require a dense W_V matrix, and their value vectors do not need to be cached across decode steps since they can be re-derived in O(1) from the token index. The paper includes a figure showing the fraction of KV cache memory saved as a function of context length, with savings becoming substantial at longer sequences.
Results
The experiments are FLOP-controlled, which is the right way to compare architectures that differ in per-token cost. Because BoV uses fewer FLOPs per token, it processes more tokens under a fixed compute budget, which the authors acknowledge as a partial contributor to its gains. Key results include:
- BoV reduces validation bits-per-byte loss relative to standard attention at both 135M and 780M scales on a held-out ClimbMix split.
- At 780M, BoV raises the average score across 21 DCLM CORE benchmarks from 0.260 to 0.272.
- Gains are largest on reading comprehension tasks (SQuAD, CoQA) and symbolic reasoning tasks, consistent with the intuition that these tasks rely less on accumulated contextual cues in the value vector.
- Compared with additive variants that keep the standard value computation and add a lookup on top, BoV performs comparably or better while using strictly less memory and compute.
- Applying context-free value vectors to all layers rather than just the last third degrades performance substantially, confirming that early layers do benefit from context-dependent value computation.
The comparison with prior methods is worth noting carefully. None of the competing approaches that add original token information to value vectors outperforms the standard attention baseline on the aggregate CORE score at 780M, suggesting limited scalability. BoV is the only variant to do so consistently.
Limitations and Open Questions
The authors are reasonably candid about what remains unresolved. The experiments top out at 780M parameters, which is small by current standards. Whether the depth-dependent pattern holds at 7B or 70B scale is genuinely unknown, and the FLOP budget used here (roughly 4 x 10^19) is well below what modern frontier models consume. Extrapolating these findings to that regime requires caution.
There is also no mechanistic account of why deeper layers exhibit this property. The paper observes the pattern empirically across ablations, but does not explain what the residual stream is doing to value vectors in early layers that makes context-dependent computation useful there, or why that changes with depth. This is not a criticism so much as an acknowledgement that the interpretability story is incomplete. Understanding the depth-dependence of value vector computation would be a genuinely interesting direction for mechanistic interpretability research.
The memory benefit also has a breakeven point. The value table has size O(|V| d) per target layer, which is a fixed cost independent of context length. The KV cache saving only materialises beyond a certain sequence length, making BoV most attractive in long-context inference settings. For short-context applications the trade-off is less clear.
One broader implication worth considering: if context-free value vectors are optimal in deep layers, this suggests that the information these layers need from the residual stream is primarily about token identity rather than accumulated context. That is a non-trivial claim about how information is organised across transformer depth, and it connects to ongoing work on residual stream geometry and the superposition hypothesis. The BoV paper does not pursue this connection, but it sits naturally in that broader conversation.
The full paper, including architecture and training code, is available at arXiv:2606.02780. The implementation is open-sourced as part of the nanochat codebase.