Mixture-of-Experts (MoE) architectures have become the dominant approach for scaling large language models without proportionally scaling compute per token. Models like DeepSeek-V2, Qwen3, and GLM-4.5 all rely on sparse expert routing to keep inference tractable. But serving these models at low latency is harder than it looks, because prefill and decode have fundamentally different computational bottlenecks. A new paper, ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration, addresses this directly with a single framework that handles both phases without any retraining.
The Problem with Existing Approaches
Most training-free MoE acceleration methods fall into one of two camps. Token-wise sparsification methods (Dynamic-MoE, NAEE, MC-MoE) reduce how many experts each token routes to, which helps prefill throughput. Expert-set consolidation methods (REAP, Lynx, SERE) shrink the set of experts active across a batch, which helps decode by reducing memory traffic. Neither family addresses both phases simultaneously, and more critically, neither family handles the excluded experts' contributions in a principled way.
Pruning and skipping simply discard those contributions. Static merging bakes excluded experts into a compressed model that cannot adapt to inference-time budgets. Similarity-based re-routing substitutes a nearby expert but never corrects for the magnitude difference between the substitute and the original. The result is that approximation error compounds precisely when budgets are tightest and the methods are pushed hardest.
This is the core insight motivating ExFold: the question is not just which experts to run, but what to do with the ones you skip.
Key Contributions and Methodology
ExFold introduces Expert Folding, a mechanism that projects excluded expert contributions onto retained experts rather than discarding them. The approach rests on two empirical observations: many expert pairs produce directionally aligned outputs, but their magnitudes differ substantially. Directional alignment makes substitution feasible; magnitude mismatch explains why naive re-routing fails and motivates a scalar correction term.
The calibration procedure is lightweight. A single frozen forward pass over unlabeled text produces two per-layer matrices: a scalar-projector matrix that captures the magnitude relationship between expert pairs, and a projection-loss matrix that records how well each excluded expert can be approximated by each retained expert. At inference time, the loss matrix routes each excluded expert to its minimum-loss target, and the scalar matrix folds the excluded contribution into that target's router weight. This folding operation adds negligible storage overhead (roughly 6 MB for the expert-scalar variant across all layers of Qwen3-30B-A3B) and is absorbed into existing router metadata rather than requiring separate computation.
The unification across phases is elegant. Prefill acceleration becomes token-level Top-K folding: select the K dominant experts per token and fold the rest. Decode acceleration becomes batch-level expert-pool folding: select the D most-activated experts across the batch and fold the remainder. Both phases share the same scalar matrix, loss matrix, and folding operator. The only difference is the selection criterion, not the recovery mechanism.
The retained-expert selection criterion itself is worth noting. Rather than ranking by activation frequency alone or by projection quality alone, ExFold uses the product of both signals. Ablations in Table 5 confirm this composite criterion consistently outperforms either component individually across benchmarks.
Implementation is as a plug-and-play plugin for vLLM, with a custom CUDA kernel for the folding operation that slots into the existing FusedMoE kernel path.
Results
The headline numbers are compelling. On Qwen3-30B-A3B with Top-4 prefill and 64-expert decode pool (from a base of 128 routed experts with Top-8 routing):
- Up to 1.41x TTFT speedup in online serving
- Up to 2.45x TPOT speedup in online serving
- Approximately 99% average quality retention across MATH 500, IFEval, GPQA, Eval+, and MMLU-Pro
The comparison against baselines is instructive. For decode-only acceleration, REAP with a 64-expert pool retains only 81.9% of original average quality on GLM-4.5-Air. ExFold with the same 64-expert budget retains 98.4%. For joint prefill and decode acceleration, naive Top-4 expert selection across both phases drops to 86.4% quality retention on Qwen3-30B-A3B, while ExFold P4+D64 retains 98.3%.
The projector ablation (Table 3) is one of the more useful parts of the paper. It compares global scalar, layer scalar, expert scalar, diagonal, and low-rank projectors. The expert scalar variant sits at a practical sweet spot: it outperforms simpler global and layer scalars on most benchmarks, costs only 6 MB of additional state, and requires no extra FLOPs beyond what the FusedMoE kernel already handles. Diagonal and low-rank projectors offer marginal quality gains in some settings but require unfused kernels and storage measured in hundreds of megabytes to gigabytes, which undermines the practical case for them.
Cross-architecture generalization holds across GLM-4.5-Air, DeepSeek-V2-Lite, and Qwen3.5-35B-A3B, which span meaningfully different expert counts, hidden sizes, and routing configurations. The calibration corpus ablation (Table 4) shows that a mixed pretraining corpus slightly outperforms domain-specific corpora, but the differences are small enough that calibration data choice is not a critical sensitivity.
Limitations and Open Questions
A few caveats are worth raising. The scalar projector is a strong simplification: it assumes the relationship between an excluded expert's contribution and a retained expert's output can be captured by a single scalar per expert pair. For most of the benchmark suite this holds well, but the assumption will break down when expert outputs are not well-aligned directionally, which the paper acknowledges as the condition under which folding is less effective. The projection-loss matrix provides a signal for when this occurs, but there is no explicit fallback for expert pairs with high loss.
The speedup numbers also depend heavily on the serving regime. The 2.45x TPOT figure is achieved under conditions where decode is genuinely memory-bandwidth-bound, which is the common case for small-batch or interactive serving. High-throughput batch inference may see smaller relative gains, and the paper's offline throughput results (Figure 5c) are more modest than the online latency improvements.
It would also be useful to see analysis of which expert pairs consistently have high projection loss, and whether these correlate with task type or input distribution. This would clarify the practical boundaries of the approach and inform calibration corpus design.
The broader implication is significant. ExFold demonstrates that recovering excluded expert contributions, rather than simply discarding them, is both technically tractable and practically valuable. The scalar projector is a minimal form of this recovery; more expressive projectors (diagonal, low-rank) exist and could be worth revisiting as kernel support improves. The framework is also explicitly complementary to kernel, scheduling, and parallelism optimizations, so it can be layered on top of other serving improvements rather than competing with them.
For practitioners running MoE models in production with latency constraints, ExFold is worth evaluating seriously. The calibration cost is a single forward pass over unlabeled data, the storage overhead is negligible, and the quality-speed trade-off is substantially better than expert dropping at matched budgets.
The code is available at github.com/Time-Rune/ExFold-MoE and the full paper is on arXiv.