As LLMs get deployed in educational contexts, one recurring assumption is that they can reason about the properties of assessment items, not just generate plausible text. A new paper from Wang et al. (arXiv:2607.28634) puts that assumption under direct empirical pressure, asking whether LLMs can accurately predict item difficulty levels for a large-scale Reading and Writing test. The results are sobering, and carry real implications for anyone building automated item generation pipelines on top of these models.
The Problem and Why It Matters
Item difficulty estimation sits at the heart of both formative and summative assessment design. In classical test theory and item response theory, difficulty parameters are estimated from empirical response data, which requires administering items to real test-takers. That process is expensive and slow. If LLMs could reliably predict difficulty prior to field testing, the economics of item development would shift considerably. The appeal is obvious, but the question of whether LLMs actually understand difficulty in any meaningful sense has received surprisingly little rigorous attention.
The study addresses this gap by treating difficulty prediction as a classification task, comparing LLMs against encoder-only transformer models and traditional feature-based supervised learning approaches. This is a sensible experimental design because it situates LLM performance within a broader model taxonomy rather than treating the comparison as LLMs versus nothing.
Methodology
The authors tested multiple LLMs across a range of prompting strategies, including zero-shot, few-shot, and chain-of-thought variants, alongside parameter sweeps over temperature settings. The primary evaluation metric is quadratic weighted kappa (QWK), which penalises predictions proportionally to how far they deviate from the true difficulty level. This is appropriate for ordinal classification tasks and preferable to raw accuracy when the cost of misclassification scales with distance.
The comparison models are worth noting. ConvBERT is an encoder-only architecture that processes full item text bidirectionally, and the feature-based supervised models use hand-crafted linguistic and psychometric features. These represent genuinely different inductive biases: the encoder models learn dense contextual representations, while the feature-based approaches encode prior domain knowledge explicitly.
Embedding analysis was also conducted, using dimensionality reduction to visualise whether item representations cluster by difficulty level. This provides a complementary geometric perspective on whether the models are encoding difficulty-relevant structure at all.
Key Findings
- Zero-shot GPT-4.1 at temperature 0 achieved the best LLM performance, with a QWK of 0.578.
- ConvBERT outperformed all LLMs, reaching a QWK of 0.625, and itself outperformed the best feature-based model.
- All LLMs struggled disproportionately with hard items, consistently underestimating their difficulty.
- GPT-5.4, despite being a more capable model by general benchmarks, showed a stronger tendency to rate items as easy, suggesting that increased model capability may actually worsen calibration on this task.
- Embedding visualisations showed item representations from different difficulty levels overlapping substantially, indicating that semantic content alone is likely insufficient to distinguish difficulty.
The finding that GPT-5.4 underestimates difficulty more than GPT-4.1 is perhaps the most striking result. One plausible explanation is that as models become more capable, they find more items trivially solvable from their own perspective, and this self-referential bias corrupts their difficulty predictions. A model that can answer a hard item easily has no internal signal that the item is hard for a human test-taker at a given proficiency level. This is a structural problem, not just a prompting artefact.
Implications and Limitations
The embedding analysis deserves more attention than it might initially receive. If items from different difficulty levels cannot be separated in the model's own representational space, that is strong evidence the model lacks the internal structure needed to make reliable difficulty judgements. Difficulty in reading and writing assessment is not purely a function of surface-level text complexity; it depends on the interaction between item design, construct being measured, and the ability distribution of the target population. None of that information is present in the item text alone, which is all the LLM has access to.
This points to a broader limitation of the study: the models are being asked to make a psychometric judgement from text-only input. Human item writers calibrate difficulty through piloting, expert review panels, and accumulated experience with how specific populations respond to specific item types. Whether augmenting LLM prompts with richer contextual information about the test population or construct would close the gap with ConvBERT remains an open question.
The choice of a Reading and Writing test also constrains generalisability. Mathematics items, for instance, have difficulty properties that are more structurally determined and may be more legible to LLMs. The authors are appropriately cautious about generalising beyond their test domain, but future work across subject areas would be valuable.
There is also a question about what ConvBERT is actually learning. Its superior performance could reflect genuine sensitivity to difficulty-relevant linguistic features, or it could reflect overfitting to surface patterns in the training data that happen to correlate with difficulty labels in this specific test. Without ablation studies or cross-test generalisation experiments, it is difficult to know how transferable that advantage is.
For practitioners building automated item generation systems, the message is clear: do not assume that an LLM generating an item at a requested difficulty level is actually producing an item at that difficulty level. The evidence here suggests the model's internal representation of difficulty is poorly calibrated against empirical difficulty as measured by test-taker response data. Items generated with a target difficulty specification should still go through standard psychometric review and field testing before operational use.
The paper is a useful corrective to optimistic assumptions about LLM capabilities in psychometric contexts. The full paper, including detailed results tables and embedding visualisations, is available at arxiv.org/abs/2607.28634.