Mixture-of-Experts (MoE) large language models present a curious deployment paradox: they activate only a small fraction of parameters per token, yet require the entire weight set to be resident in memory at all times. A model like Qwen3-30B-A3B-Base activates 3B parameters per forward pass but keeps all 30B on-device. Quantization is the natural remedy, but existing methods apply bit-width assignments at coarse granularities (layer, expert, or linear block), which turns out to be a poor fit for MoE structure. The paper BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization addresses this mismatch directly, proposing a framework that operates at the level of individual spectral components rather than whole experts or layers.
The core insight is that experts within a MoE layer share a common input-output feature space. Treating each expert as an independent quantization target ignores this redundancy and wastes representational capacity. BitsMoE instead decomposes MoE layers via SVD into a shared basis (common across experts) and expert-specific spectral factors. The shared basis is kept in full precision, preserving cross-expert structure at no per-expert cost. The expert-specific factors become the fine-grained units over which bit allocation is performed.
Methodology
The framework proceeds in three stages. First, each MoE layer is factored by SVD into a shared spectral basis and per-expert residual factors. Retaining the shared basis unquantized is a deliberate design choice: it avoids compressing directions that are jointly important across all experts, which coarse-grained methods routinely over-compress.
Second, bit-widths are assigned to spectral components by solving an integer linear program (ILP). The objective function is an activation-aware reconstruction surrogate that estimates the quantization distortion induced by assigning a given bit-width to a given spectral component. The surrogate decomposes into three terms: the squared spectral energy of the component, an activation sensitivity factor derived from calibration data, and a bit-dependent distortion estimate. Crucially, the ILP is formulated per projection type (e.g., up, down, gate projections), with the bit budget distributed uniformly across layers. This keeps the problem tractable while still capturing intra-expert, direction-level heterogeneity.
Third, inference uses the shared basis as a common projection, with quantized spectral factors applied per routed expert. The authors use Gurobi to solve the ILPs, which adds some tooling dependency but is practical given that ILP solving occurs once at compression time.
It is worth distinguishing BitsMoE from two adjacent lines of work. SVD-based MoE compressors like SliceGPT or similar rank-reduction methods discard spectral components entirely, trading capacity for memory. BitsMoE does not discard anything; it quantizes components at varying precision. ILP-based mixed-precision quantizers for MoE (e.g., methods that allocate bits per expert or per layer) operate at too coarse a granularity to capture direction-level sensitivity. BitsMoE sits at the intersection: SVD for structure, ILP for allocation, but with neither rank reduction nor coarse assignment.
Results
The experiments cover five MoE models: DeepSeek-V2-Lite, Qwen3-30B-A3B-Base, Qwen3-Next-80B-A3B-Instruct, Qwen1.5-MoE-A2.7B, and Mixtral-8x7B-v0.1. Baselines include GPTQ, HQQ, MiLo, and MoEQuant. All methods compress only MoE expert weights; attention layers remain in FP16, ensuring a controlled comparison.
Key findings at 2-bit quantization:
- On Qwen3-30B-A3B-Base, BitsMoE achieves 61.91% average accuracy across seven tasks versus 34.08% for GPTQ and 52.21% for MoEQuant, a 27.83 percentage point improvement over GPTQ.
- GSM8K (multi-step mathematical reasoning) shows the starkest gains: 75.51% for BitsMoE versus 4.32% for GPTQ and 49.36% for MoEQuant on the same model. HumanEval (code generation) shows similar patterns, suggesting that the spectral allocation disproportionately protects reasoning-critical weight directions.
- Quantization speed is 12.3x faster than GPTQ, and decoding throughput increases by 1.76x on the same model.
- At 3-bit, BitsMoE is competitive but not uniformly dominant. MoEQuant and MiLo are occasionally stronger at 3 bits, which makes sense: at higher bit budgets, fine-grained allocation matters less because even coarse methods have enough bits to avoid catastrophic compression of sensitive directions.
- Sub-2-bit results on Qwen3-30B-A3B-Base show that BitsMoE retains meaningful accuracy down to 1.4 bits (52.62% GSM8K, 47.46% average), a regime where most baselines are not even reported.
The perplexity results are less consistently strong than the downstream accuracy numbers, which is a known phenomenon: perplexity and task accuracy can diverge under aggressive quantization, particularly for reasoning-heavy benchmarks. The authors acknowledge this and present both metrics.
Limitations and Open Questions
The authors are candid about several limitations. The ILP optimizes a local, diagonal approximation to the reconstruction loss. Higher-order interactions between spectral components are ignored. For most practical purposes this approximation appears sufficient, but it is a genuine constraint on theoretical guarantees.
The bit budget is distributed uniformly across layers and projection types. This is the simplest possible high-level allocation policy, and the paper itself flags adaptive layer-wise budgeting as future work. Given that sensitivity varies substantially across transformer layers (early and late layers tend to be more sensitive), this is a meaningful gap. Combining BitsMoE's fine-grained spectral allocation with a learned or sensitivity-guided inter-layer budget could plausibly improve results further.
Attention layers, activations, and the KV cache are left unquantized. This is a reasonable scope decision for a paper focused on MoE expert weights, but it means the reported memory footprints are not full-model numbers. The authors note that orthogonal methods can handle these components.
The Gurobi dependency for ILP solving is a practical consideration. Gurobi is commercially licensed (though free for academic use), and for very large models with many experts and spectral components, solve time could become non-trivial. The paper reports that quantization is 12.3x faster than GPTQ overall, but does not break out ILP solve time in detail across all model sizes.
Implications
The broader significance of BitsMoE is methodological. It demonstrates that the right decomposition of a model's weight structure can expose allocation units that are far more informative than layer or expert boundaries. The shared-basis insight is particularly clean: rather than fighting cross-expert redundancy, BitsMoE exploits it, keeping shared directions in full precision and concentrating quantization decisions where they actually matter.
For practitioners deploying large MoE models on memory-constrained hardware, the 2-bit results are practically relevant. A model that retains 75% of its GSM8K performance at 2 bits is qualitatively different from one that collapses to near-random performance, which is what GPTQ produces here. The 1.76x decoding speedup compounds the memory savings into a real inference efficiency gain.
The code and models are publicly available at github.com/zjiayu064/BitsMoE, which should allow the community to reproduce and extend these results. Given the rapid proliferation of MoE architectures in frontier models, structure-aware quantization methods of this kind are likely to see significant follow-on work.