Backpropagation is so deeply embedded in how we train neural networks that questioning it feels almost perverse. Yet the memory overhead it imposes is a genuine barrier: fine-tuning a 7B parameter model typically requires two to three times the memory of running inference, because the gradient graph must cache activations at every layer throughout the forward pass. This rules out consumer GPUs for most serious adaptation work, and creates throughput bottlenecks even at scale. A new preprint, Forward-Pass-Only MLP Training (FPO), proposes a method that sidesteps this entirely, adapting late transformer layers without constructing an autograd graph at any point.
The result is 2.7 to 3.2 times the throughput of standard supervised fine-tuning, roughly 40% less peak memory, and, perhaps most surprisingly, cleaner preservation of off-domain benchmark performance than full-network fine-tuning achieves.
The Core Observation
FPO rests on a single empirical finding that the authors are careful to frame as an observation rather than a theorem: at late layers of a transformer, the output-layer prediction error is a reasonable approximation of the true gradient with respect to MLP weights. Specifically, the cosine similarity between the output-projected pseudogradient and the true MLP gradient falls in the range 0.47 to 0.59 across the last 25% of layers in six public models spanning 3B to 8B parameters.
That range is modest by the standards of exact gradient methods, but it is directionally consistent and architecturally stable. The per-layer cosine curve follows a characteristic pattern: low and noisy through most of the network, then rising sharply over the final quarter of layers and plateauing. This shape is consistent across all six surveyed architectures, which gives the authors grounds for a practical diagnostic.
The diagnostic itself takes approximately two minutes on a single GPU. It computes, per layer, the cosine between the output-projected signal and the true gradient, and identifies which layers sit in the high-plateau region where the approximation is strong enough to exploit. This is a meaningful contribution in its own right: rather than applying FPO uniformly, practitioners can screen any autoregressive transformer and select target layers empirically. The cosine ordering across models also correlates with experimental outcomes, so the diagnostic has predictive value beyond layer selection.
How FPO Works
The algorithm is straightforward to describe, which is part of its appeal. For each training step:
- A standard forward pass is run, caching MLP intermediate activations at target layers only.
- The output-layer error is computed as the difference between the softmax output and the one-hot target.
- This error is projected back into residual-stream space via the transpose of the language model head, then composed with the closed-form Jacobian of the final RMSNorm (and any per-layer post-feedforward norm where applicable).
- Weight updates for the down-projection at each target layer are computed directly from this signal and the cached activations.
- For the gate and up projections, a fixed feedback matrix is needed to propagate the error into MLP hidden space. FPO caches the initial down-projection weights at the start of training and uses them as this feedback matrix throughout adaptation.
No autograd graph is constructed at any point. The peak memory overhead over inference consists of the cached forward activations at target layers, optimizer state for those layers (roughly 10% of parameters), and float32 master weight copies. All three are small relative to what a standard backward pass requires.
The use of a fixed initial feedback matrix for the gate and up projection updates has a conceptual relationship to direct feedback alignment (DFA), which also uses fixed matrices in the backward path. The authors are precise about the distinction: DFA's claim is that random feedback works despite no relationship to the true Jacobian. FPO's claim is the opposite, that an initialized matrix stays close to the live Jacobian under bounded weight movement. The gradient clipping regime (relative clipping at 1%) is designed to keep cumulative drift in the down-projection weights small enough that this approximation remains valid.
Results
Experiments run across three model families: OLMo-2-7B, Qwen3-8B, and Falcon3-7B. The headline results hold consistently across all three:
- FPO achieves in-domain perplexity improvement on adaptation data in all cases.
- MMLU, ARC-Challenge, HellaSwag, and Winogrande remain within seed-noise of the pre-adaptation baseline across all three models.
- Full-network SFT and LoRA-16 do not reliably reproduce this off-domain preservation.
- A fourth condition, SFT restricted to the same target layers as FPO (SFT-partial), also preserves off-domain benchmarks, confirming that the preservation is a property of late-layer-only adaptation rather than of the FPO update rule specifically. However, SFT-partial costs 2.2 times the wall-clock time of FPO, making it impractical as an alternative.
The benchmark preservation finding deserves emphasis. It is common to treat off-domain degradation as an acceptable cost of adaptation, or to assume it is manageable with careful learning rate tuning. The FPO results suggest that the layer locality of updates matters independently of the update magnitude: restricting adaptation to late layers produces qualitatively different behaviour on off-domain tasks, not just quantitatively milder forgetting.
Limitations and Open Questions
The authors are candid about scope. FPO is validated on continued pretraining with next-token cross-entropy. Extending to instruction tuning, reinforcement learning from human feedback, or preference optimization requires re-deriving the output-projected pseudogradient under those loss functions, which is non-trivial. The method has not been tested on mixture-of-experts architectures, encoder-decoder models, or non-causal transformers. Novel normalization layouts require re-deriving the Jacobian compositions documented in the paper's appendix.
The learning rate optima vary by a factor of 20 across the three evaluated models, which is a practical concern for deployment. The authors offer a qualitative heuristic based on the ratio of gradient and activation norms to the weight norm, but no closed-form scaling rule. A practitioner extending FPO to a new architecture will need to run a sweep, albeit within a bounded window.
The cosine similarity range of 0.47 to 0.59 is also worth scrutinizing. These values indicate that the pseudogradient captures the dominant direction of the true gradient but misses a meaningful fraction of its structure. For adaptation tasks where precision matters more than throughput, this approximation quality may be insufficient. The method is best understood as occupying a specific operating point: high throughput, low memory, acceptable in-domain gain, strong off-domain preservation. It is not a drop-in replacement for full fine-tuning in all regimes.
What makes FPO interesting beyond its immediate results is the diagnostic framework. The two-minute layer viability screen is a reusable tool that could inform other selective adaptation methods, and the empirical finding that late-layer gradient alignment is architecturally consistent across six different model families is a useful characterisation of transformer behaviour that the community did not previously have in this form.
For anyone working on adaptation under memory or throughput constraints, this is worth reading carefully. The full paper is available at arxiv.org/abs/2608.14563.