The tension between computational efficiency and long-range memory has defined much of the recent debate in sequence modeling. Transformers handle long contexts well because their key-value cache grows with sequence length, giving every token direct access to the full history. Recurrent models compress that history into a fixed-size state, enabling constant-memory decoding but at the cost of recall fidelity. Once an earlier association has been overwritten by subsequent state updates, there is no mechanism to recover it. This paper, MARCH: Scaling Recurrent Memory with Content-Routed State Anchors, proposes a practical architectural solution to this bottleneck that does not require modifying the underlying recurrence.
The Core Problem
Recurrent models like Mamba, DeltaNet, and Gated DeltaNet have made substantial progress on short-context benchmarks, sometimes matching softmax attention. The problem surfaces as context length grows. These models maintain a single evolving state matrix. Each new token writes into that shared state, which means older associations are progressively diluted or overwritten. Selective forgetting mechanisms, such as the delta-rule corrections in DeltaNet, help with state tracking but do not resolve the fundamental issue: only the latest version of the state is ever available for reading.
Prior attempts to address this fall into two broad camps. One group increases per-step memory capacity through sparse states, routed mixtures, or partitioned writes. Another preserves temporal structure through hierarchical merging or logarithmic state schedules. MARCH belongs to the second camp but takes a distinctly different approach: rather than restructuring the recurrence itself, it simply checkpoints the recurrent state at periodic intervals and learns to retrieve from those checkpoints based on content.
Key Contributions and Methodology
The MARCH architecture introduces three tightly coupled components:
- State anchors. At regular intervals, the cumulative recurrent state is saved as a checkpoint. These anchors accumulate across the context, forming a bank that grows with sequence length. This directly addresses the fixed-state bottleneck without altering the recurrence dynamics between checkpoints.
- Content-conditioned anchor keys. Each checkpoint is paired with a shared learned anchor token. When the model processes this token, its hidden representation produces a compact routing key for that anchor. This is the critical design choice: rather than indexing anchors by position or temporal scale, the model learns to describe the content of each historical state in a retrievable form.
- Softmax routing with a null option. At each text token, a routing query attends over all causally available anchor keys using standard softmax attention. A learned null route allows the model to suppress the historical branch entirely when the current recurrent state is sufficient. The weighted combination of visible anchor states is read using the token's recurrent query, and the result is added residually to the current-state readout.
The design is end-to-end differentiable and trained from scratch with a standard language-modelling objective. Crucially, the underlying recurrence is untouched. MARCH wraps around Gated DeltaNet in the experiments, augmenting every recurrent layer with the anchor bank and routing mechanism. The number of anchors scales with context length, giving a controllable trade-off between memory resolution and storage cost via checkpoint density.
One subtle but important detail is the residual fusion strategy. By adding the historical readout to the current-state readout rather than replacing it, the model retains its native recurrent processing path. This means that in contexts where historical retrieval is unnecessary, the routing mechanism simply learns to weight the null route heavily, and the base recurrent behaviour is preserved. This is a sensible inductive bias that should help with training stability.
Results
The evaluation covers a broad range of tasks. On eight zero-shot commonsense reasoning benchmarks, MARCH achieves an average accuracy of 41.5%, compared to 40.1% for Gated DeltaNet and 40.0% for the Log-Linear variant. These are modest but consistent gains on tasks where long-range memory is not the primary bottleneck.
The more compelling results come from long-context evaluations. On LongBench, which tests single-document QA, multi-document QA, summarisation, and few-shot learning, MARCH consistently outperforms the recurrent baselines. On RULER retrieval tasks at 4K, 8K, and 16K context lengths, the gains are more pronounced, which is expected given that RULER specifically stresses needle-in-a-haystack retrieval. The paper also reports strong in-context retrieval results on SQuAD, TriviaQA, Natural Questions, and DROP, following the evaluation protocol of prior work on dynamic state models.
A particularly useful result is the extrapolation behaviour. MARCH shows robust performance beyond its training context length, which is a known weakness of many recurrent architectures. The anchor bank grows naturally with context, so longer sequences simply accumulate more checkpoints rather than requiring any architectural change.
Limitations and Broader Implications
The authors are candid about the main limitations. Fixed-interval checkpointing is a blunt instrument. Recurrent states do not evolve uniformly: some token spans cause large state updates while others barely perturb it. Periodic anchoring will produce redundant checkpoints in stable regions and miss important transitions in rapidly changing ones. Adaptive anchoring based on state novelty or update magnitude is the obvious next step, and the authors acknowledge this directly.
There is also a question of anchor key quality. The routing mechanism depends on the anchor token's hidden representation capturing something semantically meaningful about the state at that checkpoint. Whether a single shared learned token can reliably produce discriminative keys across diverse content types is not fully analysed. Ablations on checkpoint density and sparse routing are reported, but a more detailed analysis of routing behaviour, such as what kinds of queries actually retrieve historical anchors versus using the null route, would strengthen confidence in the mechanism.
The memory cost deserves attention. The anchor bank grows linearly with context length, which is the same asymptotic cost as a transformer key-value cache. The constant factor is much smaller because each anchor is a full recurrent state matrix rather than per-token key-value pairs, but for very long contexts this could still become significant. The paper frames this as a controllable trade-off via checkpoint density, which is fair, but practitioners would benefit from concrete memory budgets at different context lengths.
More broadly, MARCH represents a coherent design principle: rather than trying to make a single recurrent state remember everything, preserve snapshots of earlier states and learn to route to them. This is conceptually similar to episodic memory in cognitive architectures, and it connects to a growing body of work on augmenting recurrent models with external or hierarchical memory. The specific contribution here is the content-conditioned routing over state checkpoints, which is simpler than many competing approaches and appears to work well in practice.
For researchers working on efficient sequence models, MARCH is worth studying closely. It demonstrates that meaningful long-context gains are achievable without restructuring the underlying recurrence, and the architecture is modular enough to wrap around other recurrent backbones beyond Gated DeltaNet. The full paper is available at arxiv.org/abs/2608.12435.