When an LLM tells you it is 90% confident, how often is it actually right? This question sits at the heart of a new preregistered study available at arXiv:2605.23909, which evaluates confidence calibration across 11 large language models spanning both reasoning and standard chat architectures. The findings are sobering: models are systematically overconfident, and that overconfidence is worst precisely when you most need reliable uncertainty estimates, on difficult tasks.
Miscalibration is not merely a technical inconvenience. If a model reports high confidence on outputs that are frequently wrong, users cannot meaningfully distinguish reliable from unreliable responses. This matters for autonomous systems, for high-stakes decision support, and for the basic epistemic contract between an AI assistant and its user. The paper frames this clearly: hallucination and miscalibration are two sides of the same coin, both undermining the trustworthiness of AI-generated information.
Key Contributions
The study makes two distinct contributions. First, it provides a systematic, cross-model analysis of confidence calibration using five existing question sets covering true/false trivia (BoolQ), scientific multiple choice (SciQ), logical reasoning (LSAT-AR), reading comprehension (SAT-EN), and hallucination detection (HaluEval-QA). Second, and more originally, it introduces LifeEval, a new benchmark designed to provide a continuous, bias-free manipulation of task difficulty.
LifeEval works by asking models to estimate the age at death of a person given their sex and a minimum age, then state their confidence that the true age falls within a specified radius. Because the ground-truth probabilities are derived from actuarial tables, the researchers can directly compare stated confidence against empirical probability. Crucially, difficulty scales monotonically with the radius: a narrow radius is genuinely harder, and that difficulty is quantifiable rather than subjectively assigned. This sidesteps a persistent methodological problem in calibration research, where task difficulty is typically rated by humans or by the models themselves, both of which introduce their own biases.
Methodology
The 11 models tested include five reasoning models (DeepSeek-R1, Gemini 2.5 Pro, GPT-o3, Claude Sonnet 4, Claude Sonnet 3.7) and six chat models (DeepSeek-V3, Gemini 2.5 Flash, GPT-4o, Claude Haiku 3, Llama 3.1 8B, Llama 3.1 70B). All models were evaluated under identical conditions using one-shot prompting with chain-of-thought where applicable, and JSON-formatted outputs. For multiple choice questions, models were asked to assign probabilities to each answer option, not just the selected answer, which allows inspection of the full confidence distribution rather than a single scalar.
Calibration was quantified primarily through Expected Calibration Error (ECE) and a direct overconfidence measure (mean stated confidence minus mean accuracy). The researchers also ran a regression of overconfidence on question difficulty to estimate the hard-easy effect coefficient for each model.
Results
The aggregate picture is clear. Across all question sets, models reported 88% mean confidence while achieving 79% accuracy, a 9 percentage point overconfidence gap. But the more telling pattern is in how this gap varies with difficulty:
- On the hardest LifeEval tasks (narrowest radius), models reported 34.2% confidence against a true probability of only 9.6%, a massive overconfidence of roughly 25 percentage points.
- On the easiest tasks (widest radius), models reported 80.5% confidence against a true probability of 92.0%, showing substantial underconfidence.
- LSAT-AR and HaluEval produced the greatest overconfidence among existing benchmarks; SciQ and SAT-EN produced consistent underconfidence.
This pattern is precisely the hard-easy effect documented in human psychology since Lichtenstein and Fischhoff's work in the 1970s. The mechanistic explanation is a regression-to-the-mean artifact: accuracy responds more strongly to changes in difficulty than confidence does, so as tasks get harder, the gap between them widens in the overconfidence direction.
Reasoning models outperform chat models on calibration by a considerable margin. Mean ECE for reasoning models on LifeEval was 0.037, compared to 0.150 for chat models. Reasoning models also showed substantially less rounding behaviour: 61.1% of their stated confidences were multiples of 5%, versus 91.4% for chat models (with some chat models rounding 100% of responses). The correlation between stated confidence and actual probability averaged 0.94 for reasoning models and only 0.48 for chat models. The authors suggest this mirrors a broader finding: models that engage in more explicit step-by-step reasoning produce more faithful uncertainty estimates, perhaps because the reasoning process surfaces relevant unknowns.
Implications and Limitations
The parallel between human and AI overconfidence is not just descriptively interesting. It has practical implications for debiasing. In humans, prompting reflection on what one does not know is one of the more reliable ways to reduce overconfidence. The better calibration of reasoning models offers a natural experiment suggesting that encouraging models to reason through their uncertainty, rather than simply assert it, could improve calibration further. Whether this can be achieved through prompting alone or requires changes to training is an open question.
The discussion of RLHF is worth flagging. If human raters prefer confident-sounding responses, reinforcement learning from human feedback may actively reward overconfidence. This creates a structural tension between user preference and epistemic accuracy that will not resolve itself without deliberate intervention in the training pipeline.
The paper is honest about its limitations. LifeEval is narrow in content and format, which aids internal comparability but limits generalisation. Models may have encountered the underlying actuarial tables during training, potentially inflating both accuracy and confidence in ways that are hard to fully disentangle, though the authors conduct a contamination analysis in an appendix. The study also uses item-level confidence measures that conflate overestimation (thinking you are more accurate than you are) with overprecision (being too certain of your estimate), a distinction the authors flag as important for future work.
One methodological detail worth noting is the comparison between stated confidence and token-level probabilities for models where both are available. Stated confidence generally showed slightly better calibration than token probabilities, and the two distributions have quite different shapes: token probabilities almost never round to multiples of 5%, while stated confidence does so overwhelmingly. This suggests that verbally elicited confidence and implicit token-level uncertainty are measuring related but distinct things, and the choice of elicitation method matters for calibration research.
Overall, this is a carefully executed study that advances the calibration literature in two concrete ways: a systematic cross-model comparison with preregistration, and a methodologically cleaner benchmark for difficulty manipulation. The finding that the hard-easy effect transfers from human psychology to LLMs is a meaningful contribution, both for understanding why miscalibration occurs and for pointing toward remedies. The full paper is available at arXiv:2605.23909.