Multimodal Emotion Recognition in Conversations (MERC) has made considerable progress by combining audio, visual, and textual signals. Most of that progress, however, assumes all modalities are present at inference time. Real deployments rarely satisfy that assumption: network dropouts, sensor failures, and user behaviour regularly produce incomplete inputs. The performance cliff when even one modality is missing is steep, and existing remedies are only partial. This paper, C²MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning, takes a principled information-theoretic angle on the problem and proposes a framework that explicitly models both what modalities share and what makes each one unique.
The Core Problem with Existing Approaches
The dominant strategy for handling missing modalities is cross-modal consistency learning: train an encoder-decoder to recover missing representations by maximising mutual information between observed and absent modalities. The intuition is sound. If two modalities are predictable from each other, you can reconstruct one from the other. But mutual information maximisation conflates two distinct phenomena. Some cross-modal overlap is genuinely shared semantics, the kind you want to propagate during imputation. Some of it is modality-specific information that happens to correlate with another modality in the training set but carries unique task-relevant signal.
The tension is mathematically precise. Maximising consistency requires minimising conditional entropy H(Zi | Zj), pushing representations toward mutual predictability. Preserving complementarity requires maximising that same quantity, so that modality-unique cues are not collapsed away. Prior methods optimise one side of this trade-off and ignore the other. The result is reconstructed features that are consistent but informationally impoverished, losing the modality-specific signal that distinguishes, say, prosodic frustration from lexically neutral speech.
Key Contributions of C²MOE
The framework addresses the consistency-complementarity conflict by decoupling the two objectives into separate expert networks within a Mixture of Experts (MoE) architecture. The main contributions are:
- Information-theoretic decomposition: Multimodal knowledge is factorised into consistency and complementarity components. The paper formalises this through a region-based information diagram, distinguishing shared task-relevant regions (R4), modality-specific task-relevant regions (R2, R6), shared but task-irrelevant regions (R3), and purely modality-specific noise (R1, R7).
- Dual-branch imputation: A consistency branch minimises uncertainty to align imputed features with the joint distribution. A complementarity branch maximises conditional entropy to retain modality-unique cues. These run in parallel rather than competing within a single objective.
- Adaptive gating: A learnable reweighting module assigns importance scores to each expert's output dynamically, conditioned on which modalities are available and on the semantic context of the input. This avoids hard routing decisions that would be brittle under heterogeneous missing patterns.
- Unified framework: Representation learning and missing modality imputation are handled within the same formulation rather than as sequential stages, which is architecturally cleaner and reduces error propagation.
Methodology in Detail
The problem is formalised with an observability indicator per modality. Available modalities form the observed set Iobs; the task is to reconstruct representations for the missing set Imiss using only Iobs. This is standard for the field, but the treatment of what gets reconstructed differs from encoder-decoder baselines like cycle-consistency autoencoders or graph neural network propagation approaches.
Within the MoE architecture, the consistency expert maximises I(Z1, Z2) and minimises conditional entropy, producing reconstructions that are close in distribution to the true joint. The complementarity expert does the opposite: it is explicitly trained to retain the information that the consistency expert would suppress. The gating network then combines their outputs. Critically, this is not a simple ensemble. The gating weights are input-dependent and learned end-to-end, so the model can shift emphasis toward consistency when modalities are highly correlated and toward complementarity when unique signal is more informative for the downstream emotion label.
Hyperparameter analysis on CMU-MOSI and CMU-MOSEI shows stable F1 performance when the consistency loss weight sits in [0.6, 0.8] and the complementarity loss weight in [0.3, 0.5]. The asymmetry is interpretable: consistency provides the structural backbone of the imputation, while complementarity acts as a regulariser against over-smoothing. Both extremes hurt. Zero complementarity weight collapses to a standard consistency model; excessive complementarity weight introduces semantic noise that disrupts alignment.
Results, Limitations, and Implications
The paper reports that C²MOE consistently outperforms state-of-the-art methods across multiple MERC benchmarks and a range of missing-modality rates. The improvement holds across diverse missing patterns rather than just the single-modality-absent case, which is the more realistic evaluation setting.
Several limitations are worth noting. First, the paper's full experimental tables are not reproduced in the available text, so the magnitude of gains relative to baselines cannot be assessed here directly. Second, the MoE architecture adds parameters and routing overhead compared to single-encoder baselines; the paper does not appear to report inference latency or parameter counts in the excerpts available, which matters for deployment. Third, the framework currently handles pairwise modality interactions. Scaling the consistency-complementarity decomposition to three or more modalities simultaneously is non-trivial, and the information-theoretic regions multiply in complexity.
The broader implication is that treating consistency and complementarity as a single axis of optimisation has been a systematic blind spot in incomplete multimodal learning. The MoE framing is a natural fit for this decomposition because expert specialisation provides a structural mechanism for holding conflicting objectives apart. This approach could transfer to other multimodal tasks beyond emotion recognition, including medical imaging with missing scan types or audio-visual speech recognition under occlusion.
For researchers working on multimodal robustness, C²MOE is a methodologically interesting step. The information-theoretic grounding makes the design choices interpretable rather than heuristic, and the dual-branch structure is straightforward to adapt. The key open question is whether the gating network learns genuinely meaningful routing or simply learns a fixed blend that happens to work on the training distribution. Probing the gate activations across different missing patterns would be a valuable analysis for follow-up work.
Full paper: arXiv:2608.04013