A persistent problem in the evaluation of large language model reasoning is that benchmark accuracy conflates two distinct questions: whether the model has formed a useful internal representation of a problem, and whether it can then produce a correct answer from that representation. A new paper, Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs, attacks this conflation directly by proposing an axiomatic evaluation framework that measures the quality of latent thought representations independently of any downstream task performance.
This matters because the field has increasingly moved toward continuous latent reasoning, replacing discrete chain-of-thought tokens with compressed vector representations. Methods like COCONUT and Soft Thinking report accuracy gains on complex benchmarks, but the question of what these representations actually encode has received almost no principled treatment. If a representation fails silently, and downstream accuracy remains unchanged, standard evaluation gives no signal. The authors document exactly this failure mode, and provide tools to detect it.
Key Contributions
The paper makes three tightly connected contributions. First, it formalises what a thought representation should do through four functional axioms:
- Causality: the representation must causally determine the model's continuation distribution, not merely correlate with it.
- Minimality: the representation should compress input-specific information while retaining what is needed to predict the output, analogous to the information bottleneck principle.
- Separability: representations for distinct problem instances should be distinguishable, both across tasks and within a single task.
- Stability: the representation should remain consistent across semantically equivalent inputs and predict the distributional uncertainty of the model's outputs.
Second, the paper defines a concrete quantitative measure for each axiom, all computable directly on a pre-trained source LLM without retraining. KL substitution error measures Causality; an information-bottleneck residual gap measures Minimality; discriminator accuracy on same-task and cross-task pairs measures Separability; and the Distributional Consistency Score (DCS), reported as AUROC, measures Stability.
Third, the framework is applied as an empirical audit across five open-weight LLMs spanning dense, sparse mixture-of-experts, reasoning-distilled, and RL-trained architectures, evaluated on all 23 tasks of Big Bench Extra Hard (BBEH).
Methodology
The candidate thought representations evaluated include last-input-token hidden states (from all layers and from the final layer only), soft thinking tokens with and without Gumbel noise, and latent thinking vectors, each tested at thinking budgets of 1, 16, 32, 64, and 128 steps. Input prompt embeddings and random vectors serve as baselines, with output embeddings providing an upper-bound reference on axes that reward information about the generated continuation.
The evaluation backbone is a frozen LLaMA-3.2-1B with a trainable projection that maps candidate representations into its token embedding space. This design keeps computational cost constant regardless of source model size, and ensures that the KL divergence in the Causality measure reflects functional substitution rather than raw cross-model transferability. Beam search with eight beams generates the output distributions over which each axiom is evaluated, providing a controlled empirical slice of the model's continuation distribution. Semantic similarity for the Stability measure is computed using Embed-Nemotron-8B.
The logical consistency, independence, and completeness of the four axioms are proved formally in the paper's appendix, which is worth noting because axiomatic frameworks in ML are often stated informally and their properties left implicit.
Results
The findings are consistent and, in several respects, striking:
- No candidate satisfies all four axioms simultaneously, across any of the five source LLMs tested.
- Cross-task separability is near saturation for almost every candidate, including the input prompt embedding. Representations reliably encode which task a question belongs to.
- Within-task separability collapses to near-random for every candidate except the output embedding. A discriminator trained to distinguish two instances of the same task performs at roughly 50% accuracy on latent thinking and soft thinking representations. This is not a probe capacity artifact; geometric analysis shows too few effective dimensions in the within-task subspace for any probe to recover the signal.
- Causality scores for all candidates are comparable to the input prompt embedding. Thought representations carry no additional causal information about the continuation distribution beyond what is already encoded in the prompt itself.
- Minimality results are mixed. Soft thinking candidates sit at or slightly above the input embedding reference; last-input-token hidden states fall below it, encoding input-specific information that the output does not use.
- Stability (DCS) is generally high for most candidates on four of the five models, but degrades with increasing thinking steps for latent thinking, and the input embedding matches or exceeds iterative candidates on Llama-70B, suggesting that distributional uncertainty is largely predictable from the question text alone.
The failure pattern is uniform across dense, reasoning-distilled, and RL-trained model families. The authors interpret this as structural rather than a function of model size or training recipe, which is a significant claim: scaling and RLHF-style training do not appear to fix the representational collapse.
Implications and Limitations
The practical implication is that current continuous thought representations are essentially sophisticated prompt encodings. They reliably capture task identity but lose the per-instance information that would be needed to reason differently about two questions in the same category. If a model answers a spatial reasoning question correctly, the latent representation is not the reason it distinguished that question from a similar one; the input embedding already contained that signal.
For researchers building new thought representation methods, the four axioms provide explicit, decomposable optimisation targets. Rather than tuning against an aggregate benchmark score and hoping that representational quality improves, a new candidate can be scored on each axis independently, and the binding constraint identified before any retraining is undertaken. This is a genuine methodological contribution, not just an evaluation curiosity.
There are real limitations worth flagging. The Stability axiom's lexical invariance sub-property is not measured here because all candidates evaluated produce identical vectors across paraphrases by construction, leaving it trivially satisfied. The audit covers only English-language reasoning tasks on BBEH, and the conclusions may not transfer to multilingual settings or generative tasks outside reasoning. Measurement cost is non-trivial: the protocol requires beam search generation and probe training, exceeding the cost of a single accuracy benchmark run. The authors position this as a one-time cost per representation family, which is reasonable, but it does raise the barrier for adoption in fast-moving applied settings.
Perhaps most importantly, all candidates audited here are extracted from pre-trained LLMs without additional training specifically targeting the axioms. The paper explicitly notes that representations satisfying all four axioms simultaneously may require purpose-built extraction trained against these functional requirements. That is the natural next step, and the framework provides the evaluation infrastructure to pursue it.
The full paper, including formal proofs and detailed per-axiom results, is available at arxiv.org/abs/2606.27378, with code at the associated project page.