Claims that large language models can monitor and report their own internal states have accumulated steadily over the past few years. Several well-publicised studies have argued that models exhibit something resembling metacognitive monitoring: detecting when their activations have been tampered with, predicting labels derived from their own hidden states, and adjusting behaviour in response to internal feedback. A new preprint, available on arXiv, subjects two of the most prominent paradigms supporting these claims to careful empirical scrutiny and finds the evidence wanting. The conclusions matter not just for questions of AI consciousness or self-awareness, but for the practical reliability of any system that is supposed to reason transparently about its own processing.
What the Paper Argues
The authors draw a sharp distinction between two things that can look identical from the outside: genuine introspection, which requires access to internal representations beyond what the input alone provides, and input-driven pattern matching, where a model exploits surface-level features of the prompt to predict its own behaviour. This distinction is borrowed directly from the human metacognition literature, which has a long history of deflating apparent self-knowledge. Nisbett and Wilson's 1977 work showed that human self-reports frequently confabulate causes; Koriat's 1997 work demonstrated that memory metacognition often reduces to familiarity cues. The authors apply the same sceptical lens to LLMs.
Their central methodological argument is that above-chance performance on a task whose labels were derived from hidden states does not establish introspection. Because every computation in a transformer is performed over hidden states by construction, the question is not whether the model's forward pass touches its own representations, but whether there is a dissociable second-order process that takes those representations as input and reports on them. Behavioural evidence alone cannot resolve this, and the paper argues that mechanistic evidence is ultimately required.
Two Paradigms, Two Confounds
The paper focuses on two experimental setups that have been taken as evidence for LLM metacognition.
The biofeedback paradigm. Ji-An et al. (2025) and Steinmetz Yalon et al. (2026) constructed tasks where models were asked to predict labels derived from their own activations via linear classifiers or PCA directions, using in-context learning examples. The authors show that these labels are largely predictable from layer-0 input embeddings alone, without any access to the model's internal state during the forward pass. Specifically:
- A linear probe trained on the concatenation of subject and counter-entity embeddings at layer 0 matches or exceeds the model's own in-context prediction accuracy for Belief Dominance cluster labels across both Llama 3-70B and Gemma 3-27B.
- When the label-to-activation mapping is decorrelated from semantic content (by training a probe on randomly relabelled data), model performance collapses to near majority-class baseline. This is the critical control: if the model were genuinely reading off its own hidden states, it should still perform above chance on the relabelled task. It does not.
The steering detection paradigm. Lindsey (2025) reported that Claude models could detect, with non-trivial accuracy, whether a concept vector had been added to their activations via activation steering. The two-way design asked models to distinguish a normal forward pass from one with a hidden-state intervention. The authors introduce a third condition they call the "gaslight" condition: a prompt-level intervention that introduces the same kind of semantic anomaly as the activation steering, but entirely through the input text. Key results:
- In the two-way setting, Llama-3.1-70B-Instruct and Gemma-3-27b-it reproduce the pattern reported for Claude, showing low false positive rates and non-trivial detection rates for hidden interventions.
- In the three-way setting, where models must distinguish input-level interventions, hidden-state interventions, and controls, both models fail to reliably separate the first two categories. Llama concentrates probability mass on the hidden intervention option regardless of actual condition; Gemma shows a strong preference for predicting control.
- Qwen2.5-72B shows essentially no detection effect in either setting.
The failure in the three-way setting is the core empirical contribution for this paradigm. A model with genuine introspective access to its own activations should be able to distinguish a textual anomaly from an activation-level one. None of the tested models can do this reliably.
Beyond Empirical Confounds: The Principled Argument
The authors make a second, more fundamental argument that is worth separating from the empirical results. Even if all the confounds were resolved and a model demonstrated privileged access, that would not be sufficient to establish introspection in the cognitive science sense. Privileged access means only that the model's predictions carry information not recoverable from the input by a third party. But because every layer of a transformer operates over hidden states, a task that requires hidden-state information does not necessarily engage any machinery distinct from ordinary processing. The asymmetry is on the observer's side, not the model's.
Strong introspection, they argue, requires a second-order process that is mechanistically dissociable from first-order processing. This is a high bar, but arguably the right one if the goal is to make claims that connect to anything meaningful in cognitive science or philosophy of mind. Behavioural paradigms, however well controlled, cannot establish this dissociability. The authors point to mechanistic interpretability work (Macar et al., 2026) as a first step toward the kind of evidence that would actually be needed.
Limitations and Open Questions
The paper is careful about what it does and does not claim. It does not argue that LLMs lack introspective capacity; it argues that current evidence is insufficient to establish that they have it. Several limitations are worth noting.
The steering detection experiments could not replicate Lindsey's original results directly because Claude is not accessible as an open-weights model. The authors work with Llama, Gemma, and Qwen variants, which may differ in relevant ways. The fact that Llama-3.1-70B actually appears more sensitive than Claude in the two-way setting is intriguing but hard to interpret without the original model.
The relabelling control for the biofeedback paradigm is elegant, but it assumes that decorrelating labels from semantics is achievable cleanly. If the randomly labelled Ethics dataset used for relabelling still retains some semantic structure that correlates with the target probe direction, the control is weakened. The authors do not discuss this possibility in detail.
There is also a question about what finetuning-based introspection results tell us. The authors explicitly set aside studies that train models to perform introspective tasks, on the grounds that finetuning may install a task-specific mechanism rather than cultivating a general capacity. This is a reasonable methodological choice, but it means the paper's conclusions apply specifically to emergent introspection in pretrained models, not to the broader question of whether introspective mechanisms can exist in principle.
For the field, the most important implication is methodological. Future studies claiming metacognitive monitoring in LLMs should include input-matched controls as a baseline, test performance under relabelled conditions to verify that results depend on hidden-state information rather than semantic structure, and ideally pair behavioural results with mechanistic evidence of a dissociable second-order computation. The human metacognition literature took decades to develop the signal-detection theoretic frameworks needed to separate genuine metacognitive sensitivity from first-order evidence. LLM metacognition research is probably at an earlier stage than the recent optimistic papers suggest.