Most large language model architectures treat tokenization as a fixed preprocessing step, mapping raw bytes to subword units before any learned computation begins. This design choice is convenient but carries real costs: tokenizer quality varies across languages and domains, subword boundaries do not align with predictive uncertainty, and the resulting units can obscure structure that byte-level representations would expose naturally. Byte-level models have been gaining ground as an alternative, with the Byte Latent Transformer (BLT) making the approach practical by grouping bytes into dynamically sized patches based on next-byte entropy. The core insight is that predictable byte sequences can be compressed into long patches, while uncertain regions warrant shorter, more granular ones.
The paper under review, EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs, asks a natural follow-up question: if entropy already controls patch granularity, why not also use it to control which model parameters are activated for each patch? The answer the authors propose is EntropyMoE, a Mixture-of-Experts architecture that routes dynamic byte patches to expert feed-forward layers using only the scalar entropy value that the patching pipeline already computes.
The Core Idea and Why It Is Non-Obvious
Standard MoE routing projects a high-dimensional hidden state into expert logits, typically through a learned linear map with tens of thousands of parameters per layer. EntropyMoE replaces this with a learned affine transformation of a single scalar, the patch entropy, yielding just 400 trainable router parameters across 25 MoE layers. The patch hidden state is deliberately excluded from expert selection but remains the input to whichever experts are chosen. Expert assignment is therefore determined by local uncertainty, while semantic and contextual information flows through expert computation unchanged.
This separation is the conceptually interesting move. Conventional hidden-state routers conflate the question of which expert to use with the full semantic content of the token or patch. EntropyMoE argues that for byte-patch models, the routing decision can be factored out cleanly: entropy tells you how uncertain this patch is, and that uncertainty is a sufficient coordinate for selecting the right computational pathway. Whether this factorisation actually holds empirically is the central question the experiments address.
It is also worth being clear about what EntropyMoE does not do. It does not allocate more active experts to harder patches. Every valid patch activates exactly two experts regardless of entropy. The adaptation to local difficulty happens at the patching stage, where high-entropy regions produce more, shorter patches. The MoE layer then routes each of those patches independently. Granularity and expert selection are separated by design.
Experimental Setup and Baselines
All experiments use BLT-1B as the backbone, continuing training from a dense checkpoint under matched data, update counts, and active parameter counts. The sparse models replace 25 dense feed-forward blocks with 8-expert MoE layers activating 2 experts per patch, expanding total parameters from 1.46B to 4.06B while holding active parameters constant at 1.46B per patch. Three sparse variants are compared against the dense baseline:
- EntropyMoE: routes using scalar patch entropy only
- Hidden-only MoE: routes using the patch hidden state, the conventional approach
- Hidden+Entropy MoE: augments hidden-state routing with an entropy residual
All sparse variants share the same dense-copy initialisation, so differences in routing performance cannot be attributed to pretraining history. The primary metric is held-out bits per byte (BPB), with downstream evaluation on PIQA, HellaSwag, ARC-Easy, ARC-Challenge, OpenBookQA, BoolQ, and MMLU.
Results: What the Numbers Actually Show
EntropyMoE achieves the lowest held-out BPB among all four models, and this advantage over hidden-state routing is reproduced under a second continuation seed. The hidden-state variants, despite matching EntropyMoE's expert capacity, remain close to the dense baseline, suggesting that conventional hidden-state routing does not improve byte-level modelling in this setting. This is a genuinely surprising result: the richer routing signal performs worse than the scalar one.
On downstream tasks, differences are small. Avg-6 accuracy varies by at most 0.36 percentage points across all controlled models. EntropyMoE shows a statistically significant improvement on HellaSwag (roughly 1 to 1.2 points over each baseline after Holm correction), but no consistent advantage elsewhere. The authors are appropriately cautious about interpreting this as a general downstream gain rather than task-specific noise.
The routing analysis is illuminating. EntropyMoE concentrates assignments heavily: averaged across layers, only 3.12 experts receive more than 1% of assignment mass, and the top two account for 89.1% of all assignments. Hidden-only MoE distributes assignments across 7.84 active experts on average and shows almost no divergence between routing distributions for low- and high-entropy patches (Jensen-Shannon divergence of 0.004). EntropyMoE's JSD between low- and high-entropy routing distributions is 0.126, more than 30 times larger. The entropy signal does organise expert exposure in a structured way, even if the resulting specialisation is concentrated rather than distributed.
Limitations and Open Questions
The authors are commendably honest about what the results do not establish. Several limitations are worth highlighting for readers considering building on this work.
First, the experiments are continuation training runs rather than full pretraining. Whether entropy routing would be equally effective or more effective when trained from scratch is unknown. The dense-copy initialisation means all models start from a point where the feed-forward weights already encode useful representations, which may favour or disfavour particular routing strategies in ways that are hard to predict.
Second, the routing concentration is striking but ambiguous. Having 89% of assignments go to two experts across 25 layers raises the question of whether the remaining six experts are doing anything useful at all. The authors note that functional expert specialisation has not been established, only entropy-conditioned routing geometry. A dead-expert analysis or ablation removing the low-utilisation experts would clarify whether the capacity is genuinely being used.
Third, the authors mention that a shuffled-entropy control was not completed. This matters because entropy is correlated with patch length and other structural properties of the byte sequence. Without a control that breaks the correspondence between entropy values and the patches they came from, it is difficult to attribute the modelling gain specifically to the entropy signal rather than to some correlated property. This is flagged honestly in the conclusion but remains an important gap.
Fourth, sparse execution remains slower than Dense BLT despite the minimal router. Dispatch, communication, and memory movement costs dominate, and the routing parameter saving does not translate to wall-clock efficiency at the scales tested. This is a practical constraint for anyone considering deployment.
The broader implication is that low-dimensional, semantically grounded signals may be underexplored as routing coordinates in MoE architectures. Token-native models have a natural high-dimensional hidden state to route from, but byte-patch models expose an explicit uncertainty signal that appears to carry more routing-relevant information than the hidden state in this setting. Whether similar signals exist in other architectures, or whether entropy is special to the byte-patch setting, is an open and interesting question.
For researchers working on tokenizer-free modelling, sparse computation, or MoE routing more generally, EntropyMoE is worth reading carefully. The result that a 400-parameter scalar router outperforms a conventional hidden-state router under matched conditions is the kind of finding that should prompt reconsideration of what routing actually needs to accomplish.