There is a quiet crisis running through AI reasoning research, and it is not about capability. It is about definition. A new position paper, arXiv:2608.12325, argues that the generative AI community has been measuring reasoning without adequately defining it, producing a body of benchmarking literature whose construct validity is, at best, unverifiable. The paper does not claim that large reasoning models (LRMs) cannot reason. It claims we do not yet have the definitional apparatus to know whether they do.
This matters because reasoning is widely regarded as a necessary precondition for artificial general intelligence. If the field cannot operationally define what it is measuring, progress toward that goal becomes difficult to interpret. The paper is a position piece, not an empirical study, but it is analytically dense and draws on a broad synthesis of the literature spanning symbolic AI, philosophy of mind, and contemporary machine learning evaluation methodology.
Key Contributions
The paper advances three core theses. First, that operational definitions should precede measurement: reasoning phenomena under investigation must be explicitly defined, and the construct validity of evaluation must be justified against those definitions. Second, that reasoning is a process of exact rule application, not an output. Learnable rules map inputs to outputs and can encompass theorems, functions, policies, and stochastic or approximate procedures. Third, that the validity of a reasoning process derives from exact rule application, independent of which rules are selected.
Concretely, the authors offer an operational definition of reasoning as the selection and application of sequences of rules acting on prior beliefs and current evidence to produce principled belief updates. This definition is expressed in natural language, mathematical notation, and pseudocode, with illustrative Python implementations provided in an accompanying repository. The authors apply their framework to special cases including logical deduction, Bayesian inference, reinforcement learning, and probabilistic next-token prediction.
A second contribution is a checklist of best practices for communicating AI reasoning research, intended to encourage domain-specific operationalization and discourage the conflation of benchmark accuracy with demonstrated reasoning capacity.
The Reasoning Zombie Problem
One of the more conceptually interesting contributions is the introduction of the reasoning zombie (r-zombie), an analogue of the philosophical p-zombie adapted to AI systems. An r-zombie is a system that superficially behaves like an autonomous reasoner but lacks valid internal reasoning mechanisms. The authors argue that imperfect r-zombies already exist in deployed AI systems, that distinguishing them from genuine reasoners is theoretically and often empirically possible, and that determining when a use case requires a true reasoner versus an r-zombie is a practical question the field has not seriously engaged with.
This framing is useful because it sharpens the stakes. The black-box architecture and natural language interface of LRMs make it genuinely difficult to distinguish valid reasoning from sophisticated pattern completion. Chain-of-thought traces, often treated as windows into model reasoning, are neither necessary nor sufficient for trustworthy explanation. Multiple studies cited in the paper demonstrate that CoT outputs can be unfaithful to internal decision-making, and the authors note that a sufficiently capable r-zombie could produce structurally convincing but semantically hollow reasoning traces.
Construct Validity and Evaluation Failures
The paper's critique of current evaluation practice is pointed and draws on well-established methodology from empirical social science. Construct validity, the degree to which a measurement actually captures the theoretical construct of interest, has historically been central to empirical fields but has received little systematic attention in machine learning. The authors identify several specific failure modes in current reasoning evaluation:
- Final-answer accuracy on question-answering benchmarks is treated as a proxy for reasoning, but accurate answers can be produced by non-reasoning mechanisms including memorisation and statistical pattern matching.
- Chain-of-thought traces are frequently used as stand-ins for internal reasoning processes, despite evidence that they may not faithfully represent model decision-making.
- Many benchmarks fail to disentangle reasoning from recall, and performance often degrades sharply under superficial perturbations such as rephrased premises or altered numerical values, suggesting brittle pattern matching rather than generalised reasoning.
- Large-scale pre-training instils extensive priors that can allow models to achieve strong benchmark performance without robust reasoning mechanisms.
These are not novel observations individually, but the paper usefully synthesises them under a single construct validity framework, which gives the critique more analytical coherence than is typical in scattered empirical critiques.
Implications and Limitations
The call to action is pragmatic: adopt operational definitions before designing evaluations, apply the proposed checklist to reasoning-related research and product communication, and build AI reasoning systems with validity by design rather than as a post-hoc consideration. The authors explicitly position their definitions as method-agnostic, compatible with symbolic, neural, and neuro-symbolic approaches, which is a sensible choice for a framework intended to serve the broader community rather than advocate for any particular technical paradigm.
The paper is candid about its limitations. It does not offer a formal philosophical treatment of reasoning in the tradition of analytic philosophy or philosophy of mind. The proposed definitions are a starting point, not a terminus. The authors acknowledge that formalising reasoning rigorously will require sustained collaboration between philosophers, mathematicians, and computer scientists, and that their mathematical and pseudocode representations, while practically useful, are not a substitute for that deeper theoretical work.
There is also a tension the paper does not fully resolve: the definition of reasoning as exact rule application sits somewhat uneasily with the probabilistic and approximate nature of the systems the paper is ostensibly trying to evaluate. The authors address this by noting that rules can pertain to stochasticity and approximation, but the relationship between exact rule application and approximate inference deserves more careful treatment than a position paper can provide.
The broader point stands, though. A field that cannot define what it is measuring cannot reliably claim to be making progress. The generative AI community has moved fast, and the definitional scaffolding has not kept pace. This paper is a credible attempt to start building it.
Read the full paper at arxiv.org/abs/2608.12325.