As context windows in large language models have expanded to 100k tokens and beyond, a natural assumption has been that longer contexts enable better, more faithful summarization. LongNovel challenges that assumption directly. The paper introduces a multi-scale bilingual benchmark designed to systematically study how hallucinations in summarization evolve as context length grows, using long-form novels as the primary test domain. The choice of novels over news or academic papers is deliberate and well-motivated: novels contain dense webs of character relationships, implicit causal reasoning, and extended event sequences that stress-test model faithfulness in ways that shorter, more structured documents simply cannot.
Existing hallucination benchmarks for long-form text have notable gaps. NOCHA and StorySumm rely on costly human annotation and do not reach the 100k-token scale. FABLES covers 100k tokens but is restricted to a single length, preventing any analysis of how performance degrades as context grows. CLIPPER introduces multi-scale evaluation but is not designed for summarization hallucination detection specifically. LongNovel fills this gap by combining scale, bilingual coverage (Chinese and English), automated construction methods, and human revision of the test set.
Benchmark Construction
The dataset is built from 29 Chinese novels ranging from 16k to 100k tokens, supplemented by chapter-level data from the BookSum dataset. Four context-length tiers are defined: S (16k), M (32k), L (64k), and XL (100k), with 600 samples in the test set across these scales.
Eight hallucination types are defined and annotated:
- Entity hallucination: incorrect pronoun reference, subject-object swaps, entity mismatch
- Numerical hallucination: inaccurate values for quantities, ages, times, or amounts
- Relational hallucination: fabricated or substituted relationships between characters
- Reverse statement: affirmative sentences flipped to negative or vice versa
- Event hallucination: verb substitution or altered event outcomes
- Timeline disorder: events presented in the wrong chronological order
- Causal chain fabrication: invented or inverted causal links between events
- Fictional events: actions or psychological states attributed to characters with no textual basis
Two complementary construction pipelines address the tension between authenticity and category balance. The first, Multi-Model Arbitration, generates summaries using GLM4-9B-chat, Qwen3-32B, and GPT-4o, then passes these through independent evaluations by GPT-4.1 and Claude before a final arbitration step using Gemini-3-Flash-Preview. This captures realistic hallucinations that current models actually produce. The second, Entity-Referenced Hallucination Construction, perturbs human-written summaries by extracting entities and using LLMs to craft targeted hallucinations according to prompt templates for each of the eight types. This ensures that rarer hallucination categories are adequately represented rather than being swamped by the more common entity and event errors that arise naturally from model outputs. Human revision of the test set provides a final quality check.
Experimental Results
The benchmark was evaluated across a range of open-source and commercial models using balanced accuracy and Matthews Correlation Coefficient (MCC) as primary metrics. The results are unambiguous: performance degrades consistently as context length increases, for both open-source and commercial systems.
- Qwen3-14B drops from 0.588 balanced accuracy at 16k to 0.517 at 100k
- Claude-4.5-Sonnet falls from 0.755 at 16k to 0.670 at 100k
- Llama-3.1-8B-Instruct collapses entirely at 64k and 100k, producing MCC scores of -1.000 due to instruction-following failures; the model begins generating novel summaries rather than performing hallucination detection
- DeepSeek-V4-flash and the voting ensemble of DeepSeek-V4, GPT-5.2-chat, and Claude achieve the strongest results at 64k and 100k, reaching balanced accuracy around 0.820-0.830
Chain-of-Thought prompting shows inconsistent effects across architectures. It lifts Claude-4.5-Sonnet's 32k accuracy from 0.665 to 0.715 but degrades DeepSeek-v3 from 0.735 to 0.685 at the same scale. This suggests that CoT reasoning does not uniformly help with long-context faithfulness evaluation and may actually introduce noise for models that are already calibrated toward direct classification.
Supervised fine-tuning (SFT) of Qwen3-32B produces the most consistent gains among open-source models, reaching 0.720 balanced accuracy at 100k compared to 0.580 for the base model. This is a meaningful result: it suggests that length extrapolation via fine-tuning is a more reliable path to long-context hallucination detection than prompting strategies alone.
RAG-based detection shows a more complex pattern. For DeepSeek-v3, chunking the context into smaller segments for retrieval actually degrades performance at 64k (accuracy drops to 0.530), because the model's baseline error rate per chunk compounds across many decisions. For GPT-5.2-chat, which has a lower baseline error rate, RAG provides a modest benefit at 100k. This is an important practical finding: RAG is not a reliable fallback for long-context hallucination detection unless the underlying model is already highly accurate at shorter scales.
Limitations and Open Questions
The authors are transparent about the scope of their evaluation. All open-source models tested have at most 32B parameters, leaving the behaviour of larger models like Llama-3.1-70B unexplored. The benchmark is also domain-specific to novels, and whether the hallucination patterns observed here transfer to other long-form genres such as legal documents, technical reports, or scientific literature remains an open question.
There is also an inherent tension in the construction methodology. The Entity-Referenced Hallucination Construction pipeline produces controlled, targeted hallucinations, but these may be more detectable than the organic hallucinations that arise from model generation, since they are systematically structured around entity perturbation. The Multi-Model Arbitration pipeline captures more naturalistic errors, but the distribution of hallucination types in that subset is less controlled. Balancing these two sources is a reasonable engineering choice, but it means the benchmark's difficulty profile is partly an artefact of construction decisions rather than purely a reflection of real-world model failure modes.
The case studies in the appendix are worth attention. They illustrate a recurring failure mode where models correctly identify that a hallucination exists but misattribute the reason, or conversely flag a genuine inference as a hallucination because they fail to track entity-attribute bindings across thousands of tokens. These are not simple retrieval failures; they reflect a deeper limitation in how current models maintain structured knowledge representations over long contexts.
Implications for the Field
LongNovel makes a concrete case that expanding context windows is a necessary but not sufficient condition for faithful long-form summarization. The benchmark's multi-scale design enables researchers to study the degradation curve directly rather than treating long-context performance as a binary capability. The bilingual coverage is also a genuine contribution: most hallucination benchmarks are English-only, and Chinese novel summarization presents distinct challenges around character name disambiguation and implicit cultural context.
For practitioners building summarization pipelines over long documents, the results suggest that ensemble voting across diverse commercial models currently outperforms any single model, and that fine-tuning for length extrapolation is more effective than prompting strategies for open-source systems. The dataset and code are available at github.com/BDML-lab/LongNovel.