A persistent tension in large language model adaptation is that the most effective approach, full fine-tuning, is prohibitively expensive, while the most efficient approach, LoRA and its variants, consistently underperforms. A new paper, FuRA: Full-Rank Adaptation with Spectral Preconditioning, argues that both approaches share a fundamental flaw: they apply weight updates without any regard for the spectral geometry that pretraining has already established. The proposed solution, a block tensor-train factorization that freezes one singular basis while training the rest, is elegant in its simplicity and surprisingly effective in practice.
The Core Diagnosis
The paper's central claim is that spectral preconditioning is the missing ingredient in current fine-tuning methods. When a model is pretrained on web-scale data, each weight matrix develops a structured singular value decomposition whose column space encodes the model's learned feature representations. Fine-tuning on a small, narrow task distribution introduces gradients that are noisy estimates of the true task-relevant directions. Allowing those noisy gradients to freely perturb the weight matrix, whether through full fine-tuning or through a low-rank adapter added on top, risks degrading the robust features that pretraining established. This is, in essence, a more mechanistic framing of catastrophic forgetting.
To validate this diagnosis before proposing a fix, the authors run a controlled experiment they call SVD FT. Each weight is decomposed as W = UΣV⊤, the left singular basis U is frozen, and only Σ and V⊤ are trained. This has the same parameter count as full fine-tuning but introduces a structural constraint: updates are confined to the column space of the pretrained weight. The result is instructive. SVD FT improves target-domain math reasoning performance relative to full fine-tuning on LLaMA-3-8B, and it also better preserves source-domain commonsense reasoning accuracy, where full fine-tuning degrades. The column-space constraint is doing real work.
The FuRA Method
SVD FT is not practical at scale. It retains a full-size trainable factor and doubles forward-pass cost. FuRA addresses this through a block tensor-train (BTT) decomposition, reparameterizing each weight matrix as W = LSR, where:
- L is a large frozen core initialised from the block-wise SVD basis of the pretrained weight
- S contains trainable per-block singular values
- R is a small trainable core
The decomposition is lossless: the BTT exactly represents the original weight at initialisation. By freezing L, every update is confined to the pretrained column space, exactly as in SVD FT, but the trainable parameter count drops to roughly 2% of the original weight at typical hidden dimensions of 4096. The L-stage reduces to a dense matrix-vector multiplication and the R-stage to a batched block matrix multiplication, both well-supported operations. After training, the three cores merge back into a single dense matrix, so there is no inference overhead.
The paper proves two formal properties of this construction. First, although only a small fraction of parameters are trained, the effective update ΔW can be full-rank, because the frozen block-wise singular bases span the full output space when taken together. Second, the effective update from training R is a column-space projection combined with singular-value preconditioning, directly analogous to the SVD FT result but achieved at LoRA-level cost. The trainable S additionally decouples magnitude and direction in the spectral domain, providing finer-grained control than a single low-rank adapter.
Results
The empirical evaluation covers three settings: supervised fine-tuning for commonsense reasoning, math reinforcement learning with verifiable rewards (RLVR using GRPO), and visual instruction tuning of a vision-language model.
- On LLaMA-3-8B commonsense SFT across eight benchmarks, FuRA averages 88.01%, compared to 86.64% for full fine-tuning, 85.04% for DoRA, and 83.46% for LoRA at rank 64
- On Qwen-1.7B and Qwen2.5-7B math RLVR, FuRA outperforms full fine-tuning on held-out benchmarks including MATH-500, AMC23, and AIME-24/25
- On VLM visual instruction tuning, FuRA exceeds full fine-tuning by 1.1 points
- QFuRA, the 4-bit quantized variant, outperforms QLoRA by 2.51 points on LLaMA-3-70B math fine-tuning
System overhead is competitive. FuRA matches LoRA and DoRA in per-step wall-clock time (around 0.046 seconds versus 0.043 and 0.045 respectively on a single H100) and achieves the lowest peak GPU memory in the comparison at 22.7 GB, below even LoRA at 26.8 GB. Full fine-tuning requires 76.9 GB. The block-SVD initialisation is a one-time cost of roughly one minute for an 8B model.
Limitations and Open Questions
The paper is candid about what remains unresolved. The theoretical account of why column-space-confined updates improve generalisation is largely empirical at this stage. The authors demonstrate the phenomenon clearly but a rigorous explanation connecting spectral preconditioning to generalisation bounds is absent. This is not unusual for the field, but it does mean the method's behaviour in distribution-shift regimes beyond those tested is hard to predict from first principles.
The comparison with LIFT, which achieves full-rank sparse updates, is worth scrutinising. LIFT reaches 87.88% average accuracy on the commonsense benchmark, marginally below FuRA's 88.01%, but uses 6.23% trainable parameters and requires gather/scatter operations that complicate tensor-parallel training. FuRA's advantage in step time and memory is genuine, but the accuracy gap over LIFT is narrow enough that task-specific considerations could tip the balance.
The block size selection in the BTT decomposition is a design choice that the paper fixes for computational convenience. Whether the optimal block structure is task-dependent, and whether it could be learned or adapted, is an open question. The authors also note that the frozen orthonormal rows of L could in principle support more aggressive quantization than standard QLoRA, but QFuRA does not yet fully exploit this.
The broader implication of this work is that the field has been underweighting the structural properties of pretrained weights when designing adaptation methods. Rank alone is insufficient; the subspace in which updates operate matters at least as much. FuRA makes this concrete by showing that a single architectural constraint, freezing the pretrained singular basis, consistently outperforms unconstrained alternatives at the same computational budget. Whether similar preconditioning principles apply to other modalities, architectures, or decompositions beyond SVD is a natural direction for follow-on work.
Code is available at github.com/olokevin/FuRA-NIPS.