Residual connections are one of those architectural decisions that feel almost invisible until you start asking why they work as well as they do. Standard PreNorm residual streams accumulate sublayer updates with fixed unit weights, which is simple and stable but increasingly blunt as network depth grows. A stream of 48 layers worth of uniform accumulation is, by construction, discarding a lot of structure. The question this paper asks is whether you can recover some of that structure cheaply, without touching the attention or MLP modules themselves.
The paper, WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers, builds on two prior lines of work. Attention Residuals introduced content-dependent depth-wise routing, letting each layer selectively read from previous layer outputs via a softmax mixer rather than blindly summing everything. Block Attention Residuals compressed this to operate over block-level summaries rather than individual layer outputs, reducing memory and communication cost. WAV v1 sits on top of Block AttnRes and asks a pointed question: if you compress a block to its summed residual displacement, what are you throwing away?
Key Contributions
The central observation is that a block summary C_b, the sum of all sublayer updates within a block, is analogous to a DC or low-frequency component of the residual trajectory inside that block. It tells you where the block ended up, but not how it got there. Two blocks with identical sums could have very different internal dynamics: one might be attention-dominant, another MLP-dominant; one might have its significant updates early, another late.
WAV v1 addresses this by augmenting each block representation with two zero-sum directional detail bases:
- Phase basis: the difference between attention and MLP sublayer updates within a block, capturing which module dominated the block's trajectory.
- Split basis: the difference between first-half and second-half sublayer updates, capturing whether the block's activity was front-loaded or back-loaded.
Both bases are accumulated online as residual updates are produced, so there is no additional forward pass cost to compute them. They are routed by the same depth-wise softmax mixer as the block summary, but introduced with a negative initial bias and RMS-matched to the corresponding block summary. This is a conservative design choice: the detail sources are suppressed early in training and must earn their contribution. The final prediction mixer also ignores detail sources by default. The framing as multi-resolution routing is apt; the block summary is the coarse approximation, and the detail bases are corrections that expose intra-block directional structure.
Experimental Setup and Results
The experiments use character-level GPT decoder-only models on TinyStories and Text8, at depths of 12, 24, and 48 layers. Models are small (roughly 5.5M to 22M parameters) with a hidden dimension of 128, SwiGLU MLPs, and PreNorm RMSNorm. Five residual mechanisms are compared: Standard Residual, Block AttnRes, ReZero, LayerScale, and WAV v1.
The headline result is a clean depth-dependent trend:
- At 12 layers, WAV v1 is worse than Block AttnRes on both datasets. The directional detail sources appear to add noise rather than signal when the residual trajectory is short.
- At 24 layers, WAV v1 becomes competitive, slightly improving TinyStories and approximately matching Text8.
- At 48 layers, WAV v1 is the best method on both datasets. On TinyStories it reduces validation loss from 0.4960 to 0.4738 relative to Block AttnRes, a 2.19% perplexity reduction. On Text8 the improvement is 0.0057 in validation loss, a 0.57% perplexity reduction.
ReZero and LayerScale both underperform Standard Residual at deeper configurations, which is a useful sanity check: not all residual modifications scale gracefully. The depth-dependent pattern for WAV v1 is the more theoretically interesting result, since it suggests the detail bases are not simply adding capacity but are providing information that becomes useful only when the residual trajectory is long enough to have meaningful internal structure worth routing.
Limitations and Open Questions
The paper is candid about its preliminary status. There are no standard deviations across seeds, which matters because the improvements at 24 layers are small enough that single-run noise could plausibly explain them. The 48-layer TinyStories result is more convincing in magnitude, but even there, confidence intervals would substantially strengthen the claim.
The experiments are also limited to small character-level models. Character-level language modeling has different statistical properties from token-level modeling at scale: the sequence structure is different, the vocabulary is tiny (256 tokens here), and the residual stream dynamics may not transfer directly. Whether the depth-dependent benefit of WAV v1 persists at, say, 1B parameters with byte-pair encoding is an open question the paper does not attempt to answer, reasonably given its scope.
The two detail bases are hand-designed. The phase basis assumes the attention-vs-MLP split is a meaningful axis of variation; the split basis assumes temporal position within a block matters. Both are plausible priors, but there is no ablation showing that these particular bases are better than, say, random zero-sum projections of similar magnitude. The paper acknowledges this and lists soft orthogonal detail bases and learnable block wavelet bases as future directions, which is the right framing.
There is also a subtlety worth noting in the experimental design: with 4 fixed residual blocks across 12, 24, and 48 layers, the block granularity changes with depth. At 48 layers, each block contains 12 sublayers, which is a long enough trajectory that phase and split differences are likely to be meaningful. At 12 layers, each block contains only 3 sublayers. This confound between depth and block granularity makes it harder to isolate whether the benefit comes from absolute depth or from the richness of intra-block trajectories at a given block size.
Implications
The broader framing of residual routing as a multi-resolution problem is genuinely useful. Most work on residual connections treats the stream as a flat sequence of updates; thinking about it as having frequency structure, where block summaries are low-frequency and directional bases are high-frequency corrections, opens up a cleaner design vocabulary. Whether the wavelet analogy is more than metaphor remains to be seen, but it suggests a principled way to think about what information different routing mechanisms can and cannot represent.
For practitioners training deep decoder-only models, the practical takeaway is modest but real: if you are already using Block AttnRes, WAV v1 is a low-cost augmentation that appears to help at 48 layers with minimal architectural change. The conservative initialisation means it is unlikely to destabilise training, and the online accumulation of detail bases adds negligible compute. The caveat is that the benefit at shallower depths is either absent or negative, so this is not a universal drop-in improvement.
The full paper is available at arxiv.org/abs/2606.06564. Given the preliminary nature of the results, the most important next step is reproducing the 48-layer findings across multiple seeds and extending to token-level models at larger scale.