As large language models are increasingly pressed into service as simulators of human behaviour, physical processes, and economic systems, a basic question has gone largely unanswered: can they actually sample from the distributions they purport to understand? A new benchmark, UnpredictaBench, provides the most systematic answer yet, and the answer is not encouraging. The paper is available at arXiv:2606.06622.
The core problem is subtle but consequential. LLMs can reason about probability in the abstract; prior work has shown they can estimate percentiles and identify distributional structure. But reasoning about a distribution and sampling faithfully from it are different cognitive tasks. When a model is used to simulate, say, how many people might attend an event given some stochastic process, a single plausible answer is insufficient. What is required is a sequence of outputs whose empirical distribution matches the true underlying one. This is precisely what most models fail to do.
Key Contributions
The benchmark consists of 448 test instances spanning 40 target distributions across four task categories:
- Text tasks: natural language descriptions of random processes (e.g., coin flips, Poisson arrivals)
- Code tasks: stochastic programs whose output distribution must be inferred and reproduced
- RealWorld tasks: scenarios grounded in physical or computational stochasticity, such as race conditions in multithreaded code
- Shuffling tasks: list permutation problems evaluated via Lehmer code encoding
The evaluation metric, KS@N, is the paper's other main contribution. For each problem, a model generates N independent samples; these are compared against a large reference draw (M = 10,000) from the ground truth using a two-sample Kolmogorov-Smirnov test. KS@N reports the fraction of problems where the model's samples are not rejected at a p-value threshold of 0.0001. The choice of threshold is conservative, minimising false negatives while still demanding genuine distributional alignment. Crucially, increasing N makes the test harder: at N=1, every model achieves 100% by construction, since a single value within the support is almost never rejected. At N=100, the benchmark becomes a genuine stress test of calibration.
Two complementary metrics are also reported: Wasserstein Distance Z-score (WDZ), which captures systematic shifts in location and tail behaviour, and Jensen-Shannon Divergence (JSD), which measures density-level shape mismatch. The three metrics are broadly consistent across models and categories, which validates KS@N as a reliable summary statistic.
Results
The headline finding is stark. No model exceeds 33% at KS@100, and most cluster between 0% and 20%. The full spread is worth examining:
- Nemotron-3 Super 120B leads at 32.64%, nearly doubling the score of the third-ranked model
- GPT-4o (23.90%) and DeepSeek V3.2 (21.73%) form a second tier
- GPT-5.4 scores only 15.18%, behind the 2B parameter Qwen3.5-2B at 17.67%
- Claude Sonnet 4.6 achieves 97.73% at KS@2 but collapses to 5.04% at KS@100, one of the steepest degradation curves in the dataset
- Grok-4.1-fast, Mistral Large, and several Qwen MoE variants fall below 7%
The category-level breakdown reveals further structure. Shuffling tasks are the hardest, with GPT-5.4, Mercury-2, Claude Sonnet 4.6, and both large Qwen MoE models collapsing to exactly 0%. Meanwhile, smaller models such as DeepSeek V3.2, OLMo-3 7B, Llama-3.2-1B, and Qwen3.5-2B all sustain roughly 37% on shuffling tasks, suggesting that whatever property enables distributional diversity over permutation sequences is not simply a function of parameter count. The qualitative analysis reinforces this: the failure mode is visible in the model's logit distributions, not just in its samples. When logit and sample distributions align closely across independent calls, the diversity problem is a property of the model's internal beliefs, not a decoding artefact.
Instruction tuning appears to be a significant culprit. The Llama-3.2-1B base model covers distributional support substantially more broadly than its instruction-tuned counterpart on the same tasks. This is consistent with the broader literature showing that RLHF-style training penalises unusual outputs, systematically reducing the variance of generated sequences. Models heavily optimised for precise, deterministic reasoning, including Claude Sonnet 4.6 and the large Qwen MoE variants, score near the bottom across all categories.
Methodology and Design Choices
Several design decisions deserve scrutiny. The use of a fixed temperature of T=1.0 is a reasonable default, but it means the benchmark does not explore whether temperature scaling could partially recover distributional fidelity. The reasoning experiments (setting reasoning effort to "xhigh") show modest improvements but nothing close to solving the problem, which is informative: the failure is not simply one of missing explicit reasoning steps.
The Lehmer code encoding for shuffling tasks is an elegant solution to the problem of evaluating distributions over permutations with a scalar metric. Using the first Lehmer coordinate as a proxy is a deliberate choice: it has the largest support of any coordinate and therefore provides the strictest one-dimensional diagnostic. The authors acknowledge this does not fully characterise the joint permutation distribution, but it is a principled and interpretable approximation.
One limitation worth flagging is that 89% of prompts were generated by GPT-5.4 itself. This introduces potential phrasing biases and may inadvertently advantage models trained on similar synthetic data. The authors note this, but it remains an open question whether the benchmark's difficulty profile would shift with human-authored prompts. All prompts are also in English, which limits conclusions about multilingual settings.
Implications
The broader implication is uncomfortable for anyone using LLMs as simulation components. If the best available model achieves roughly one-third distributional accuracy on simplified, single-output distributions with known ground truths, the prospects for faithful simulation of complex stochastic systems such as human opinion dynamics or epidemiological spread are considerably worse. The paper makes this point explicitly: we should not expect LLMs to capture human behavioural distributions given their failure on canonical statistical ones.
The correlation between UnpredictaBench scores and utility metrics from creativity benchmarks (NoveltyBench and CREATE) is an interesting secondary finding. It suggests that distributional fidelity is not an isolated capability but tracks something more general about model quality, and that KS@N offers a statistically grounded alternative to LLM-as-a-judge evaluation for tasks where output diversity matters.
The benchmark is available on Hugging Face under CC BY 4.0, with code and ground-truth values on GitHub. Future work could extend KS@N to multivariate distributions, incorporate non-English prompts, or use the benchmark as a diagnostic for targeted fine-tuning interventions. The gap between current models and a simple pseudorandom number generator is large, and closing it will require understanding why instruction tuning so reliably suppresses the output diversity that distributional sampling demands.