Mixture-of-Experts (MoE) language models have become a practical route to scaling model capacity without proportionally increasing inference cost. By routing each token through only a small subset of expert feed-forward networks, models like DeepSeek-V2, OLMoE, and Qwen-MoE achieve strong performance while remaining computationally tractable. A natural question follows: can the routing behaviour itself be used to guide how we fine-tune these models for specific downstream tasks? This paper, TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation, argues that existing approaches answer this question poorly, and proposes a cleaner alternative.
The Problem with Aggregate Routing Statistics
Current MoE adaptation methods typically identify task-relevant experts by aggregating routing statistics over task data: which experts are selected most frequently, or which have the highest average gate scores? The intuition is reasonable, but the paper presents a compelling empirical challenge to it. Experts selected by high activation frequency include many with a negative SuccessGap, meaning they are actually activated more often on instances the base model gets wrong than on instances it gets right. Some of the most positively discriminative experts are missed entirely by frequency-based selection.
This is not a minor calibration issue. It reflects a conceptual mismatch: aggregate frequency measures usage, not association with correct task completion. A frequently activated expert might be capturing surface-level token patterns common across the task distribution, rather than the computation actually responsible for solving the task correctly.
A second limitation compounds this: even methods that identify task experts well tend to use them only to decide which experts to fine-tune or how to regularise routing. The idea that expert activations could inform where in the training signal to concentrate supervision has received little attention.
The TEXAS Framework
TEXAS addresses both limitations through two coupled components.
Correctness-conditioned expert discovery. Rather than aggregating routing statistics over the full training set, TEXAS splits training instances into those the base model solves correctly and those it fails on. It then computes, for each layer-expert pair, the difference in mean activation rate between the successful and failed partitions. Experts with a larger positive gap are more strongly associated with correct task completion and are retained as task experts. A candidate filtering step removes experts that are too rarely or too uniformly activated to provide a reliable signal.
Task-expert-aware supervision allocation. During fine-tuning, TEXAS does not restrict which experts are trained or impose a target routing distribution. Instead, it dynamically upweights the cross-entropy loss on answer tokens in base-model failed instances when those tokens' current computation paths activate the discovered task experts. Tokens in successful instances and tokens that do not activate task experts receive standard supervision weights. The result is a training signal that is more concentrated on the positions and instances where task-expert pathways are engaged but not yet producing correct outputs.
Crucially, the supervision weighting is dynamic: it is computed from the model's actual routing decisions during each forward pass, not precomputed from the base model's routing. This means the method adapts as the model's routing evolves through training.
Experimental Results
The evaluation spans three MoE architectures (DeepSeek-V2-Lite, OLMoE-1B-7B, Qwen1.5-MoE-A2.7B) and six benchmarks covering mathematical reasoning (GSM8K, MATH500), code generation (HumanEval, MBPP), general knowledge (MMLU), and instruction following (IFEval). All methods use LoRA adapters on routed experts with frozen pretrained weights outside the adapter and router parameters.
TEXAS achieves the best or tied-best result in 17 of 18 model-task settings. Selected highlights:
- Average improvement over the strongest baseline (RoMA): 1.5 points on DeepSeek, 1.5 on OLMoE, 1.3 on Qwen.
- Average improvement over standard SFT under identical trainable parameter counts: 3.0, 2.4, and 2.7 points respectively.
- GSM8K gains over RoMA of 2.6, 2.2, and 1.9 points across the three models; IFEval gains of 1.5, 2.2, and 1.9 points.
- MMLU is the exception, where TEXAS performs comparably to baselines rather than clearly surpassing them. The authors attribute this to weaker task-expert pathway strengthening on knowledge retrieval tasks, which is consistent with the pathway analysis they report.
Ablations confirm that both components contribute independently. Removing the correctness conditioning (reverting to frequency-based expert selection) and removing the supervision reweighting each degrade performance, and the full method outperforms either component alone.
Limitations and Open Questions
The paper is methodologically careful and the ablations are thorough, but several questions remain worth raising.
The binary correctness split is a strong simplification. For tasks with partial credit, graded difficulty, or multiple valid solution paths, the distinction between a successful and a failed instance is less clean. The authors acknowledge this and suggest richer outcome signals as future work, but the current method's dependence on a reliable correctness oracle is a real constraint. On tasks like MMLU, where the base model already achieves reasonable accuracy, the split may produce noisier expert discovery, which might partly explain the weaker gains there.
The discovered task experts are fixed before fine-tuning begins. As the model's routing behaviour shifts during training, the initially identified experts may become less representative. Dynamic updating of the task expert set during training is flagged as future work, and it seems like a natural extension that could matter more in longer training runs or multitask settings.
The evaluation uses relatively small MoE models (OLMoE at 1B active parameters, DeepSeek-V2-Lite and Qwen at roughly 2-3B active parameters). Whether the SuccessGap signal remains as clean and the supervision reweighting as effective at larger scales, where expert specialisation patterns may differ, is an open empirical question.
Finally, the computational overhead of TEXAS is non-trivial at the discovery stage: it requires running the base model over the training set and recording per-instance routing decisions. The paper reports this in the appendix, but for practitioners working with large training sets this is worth factoring in alongside the performance gains.
Despite these caveats, TEXAS represents a genuinely useful conceptual contribution. The move from asking "which experts are used for this task?" to "which experts are associated with getting this task right?" is simple in retrospect but has clear practical value. The further step of using those experts to shape the training signal, rather than just the trainable parameter set, opens a direction that seems underexplored relative to its potential. The paper makes a credible case that internal computation pathways are an underused source of information for guiding adaptation, and the results across diverse architectures and tasks support that case.