One of the more uncomfortable tensions in modern language model research is between efficiency and interpretability. Latent space reasoning methods, which bypass discrete token generation by feeding hidden states directly back into the model as continuous inputs, have demonstrated real gains in token efficiency. The problem is that the reasoning process becomes completely opaque. You get a faster model that you cannot interrogate. The paper Self-Explainable Latent Reasoning (SELR) takes a direct shot at this tension, proposing a training framework that produces a single model capable of both efficient latent reasoning and translating its own continuous thoughts back into human-readable text.
This matters because the two dominant prior approaches each carry significant baggage. Coconut-style models reason entirely in the latent space and offer no interpretability at all. The Heima framework adds a separate post-hoc decoder to provide explanations, but this introduces parameter overhead and, more critically, creates a structural disconnect: the model that reasons and the model that explains are different objects. There is no guarantee the explanation reflects what actually happened during inference. SELR collapses this into a single model trained with a unified objective.
Key Contributions
- A multi-task training objective combining an Answer Loss (optimising latent reasoning for correct final answers) and a CoT Loss (training the same model to decode its own latent representations into step-by-step reasoning text).
- The first application of explainable latent reasoning to Vision-Language Models (VLMs), demonstrated on Qwen2.5-VL-3B-Instruct fine-tuned on LLaVA-CoT-100k.
- A multi-stage curriculum training strategy that substantially outperforms single-stage training, particularly for LLMs.
- A faithfulness analysis showing that decoded reasoning chains correlate meaningfully with model correctness, rather than being generic post-hoc rationalisation.
Methodology
The core mechanism is straightforward in principle. During the reasoning phase, the model generates a sequence of latent thought tokens by feeding its own last hidden state back as the next input embedding, bypassing discrete vocabulary projection. These continuous tokens encode reasoning information more densely than text tokens can. During the decoding phase, those same latent thought tokens are fed back into the identical model with a decoding prompt, and the model generates human-readable reasoning steps from them. No separate architecture is required.
The training objective combines two losses. The Answer Loss is standard cross-entropy on the final answer tokens. The CoT Loss supervises the model to reproduce ground-truth reasoning steps when given its own latent thoughts as input. This dual supervision is the key design insight: the CoT Loss forces latent representations to be semantically aligned with human reasoning structure, which simultaneously addresses the supervision problem (there is no human-labelled ground truth for continuous thoughts) and the interpretability problem.
Two training regimes are explored. The single-stage method trains with fixed latent length and the sum of both losses from the start. The multi-stage method, which consistently performs better, begins with variable latent length and a Single Step Loss that aligns individual latent thoughts to individual reasoning steps, then transitions to fixed latent length with Full CoT Loss in a second stage. For VLMs, the authors retain Single Step Loss in the second stage rather than switching to Full CoT Loss, because the VLM dataset's reasoning steps are long and information-dense, and switching degrades decoding quality. This is a sensible empirical adaptation rather than a principled architectural choice, and the authors are transparent about it.
Results
The VLM results are the headline finding. Evaluated across six benchmarks (MMStar, MMBench, MMVet, MathVista, AI2D, HallusionBench) using Qwen2.5-VL-3B-Instruct as the base, SELR achieves the following:
- Average accuracy improvement of up to 0.86 percentage points over the base model, compared to a 1.92 point drop for Heima.
- Token count reduction of over 70%, from approximately 50 tokens average to around 13 tokens, with an 8.7x wall-clock inference speedup over the SFT baseline.
- Higher consistency scores between decoded reasoning and final answers than Heima on MathVista (0.2097 vs. 0.1671), suggesting the explanations are more faithful to the actual computation.
For LLMs, SELR Multi outperforms Coconut on all four math reasoning benchmarks and is competitive with CoLaR, which uses roughly twice as many latent tokens. The gap between SELR and CoT-SFT reflects a deliberate token budget difference: CoT-SFT uses over 150 tokens per question, SELR uses 6. When CoT-SFT is constrained to the same 6-token budget, its accuracy on GSM8k collapses to 10.31% versus SELR's 42.46%, which is a useful controlled comparison that contextualises the headline numbers.
The faithfulness analysis is worth dwelling on. Correct predictions produce decoded CoT with GPT-4o alignment scores around 0.5; incorrect predictions score around 0.26. This gap is meaningful. It suggests the decoded reasoning is not just a plausible-sounding confabulation but actually tracks the model's internal state to some degree. That said, scores around 0.5 for correct predictions are not high in absolute terms, and the metric itself (GPT-4o scoring alignment on a 0-1 scale) is not a ground-truth measure of faithfulness. It is suggestive evidence, not proof.
Limitations and Open Questions
Several limitations are worth flagging. The MMVet degradation for multi-stage SELR variants is attributed to the model's reduced capacity for extended token generation, which is a real constraint of latent compression. The authors note this is shared with Heima, but it remains a practical limitation for tasks requiring long-form generation or complex multi-step arithmetic.
The faithfulness question is the deepest unresolved issue. Even with the consistency metric showing correlation between decoded reasoning and answer correctness, it is not clear whether the latent representations are causally responsible for the correct answer or whether the decoding process is reconstructing plausible reasoning post-hoc. The unified architecture makes this more likely to be genuine than Heima's separate decoder, but the paper does not provide causal intervention experiments that would settle the question definitively.
The authors themselves identify the lack of a stable learned stopping mechanism as a significant challenge for future work. Currently, latent length is fixed or controlled by curriculum design rather than learned adaptively. A model that could determine how much latent reasoning a given input requires would be substantially more practical.
There is also a question of scalability. The VLM experiments use a 3B parameter model (with a 7B ablation in the appendix), and the LLM experiments use a 1B model. Whether the training dynamics and efficiency gains hold at frontier model scales remains to be demonstrated.
Despite these open questions, SELR represents a genuinely useful step. The core insight, that a model can be trained to be its own latent-to-language translator without architectural overhead, is clean and the empirical evidence supports it. For practitioners working on inference efficiency who have been reluctant to adopt latent reasoning because of the black-box problem, SELR offers a credible path forward.
The full paper is available at arxiv.org/abs/2608.13570 and the project page with additional details is at jasondayuan.github.io/SELR/.