Most RAG evaluations measure whether a model gets the right answer, not how it got there. This distinction matters more than current benchmarks acknowledge. When a model correctly answers a question given a retrieved passage, standard metrics like Exact Match or F1 cannot tell you whether the model read the passage or simply recalled the answer from pre-training. This paper, available at arXiv:2606.23695, directly addresses that ambiguity by introducing a continuous, probability-grounded evaluation metric and using it to challenge a core assumption in LLM deployment: that larger models are always better.
The Core Problem with Discrete RAG Metrics
Binary evaluation heuristics conflate two epistemically distinct behaviours. A model answering correctly from parametric memory and a model genuinely extracting from a retrieved context look identical to an EM scorer. The authors call this "epistemic blindness", and it has real consequences for how we design and select models for RAG pipelines.
The proposed solution is the Normalized Context Utilization (NCU) metric. Rather than scoring outputs as correct or incorrect, NCU computes the fractional reduction in log-probability uncertainty when a model is given an oracle context versus answering zero-shot. Formally, it measures the informational gain of the context by comparing length-normalized token log-probabilities across three conditions: zero-shot (no context), oracle (clean, relevant context), and adversarial (context containing a factually conflicting answer). Length normalization is important here because tokenization granularity varies across architectures, and raw sequence log-probabilities would systematically disadvantage models with finer-grained vocabularies.
The adversarial condition is where the metric earns its analytical value. If a model is told the answer is X in the retrieved passage but its parametric priors strongly favour Y, NCU captures whether the model follows the context or overrides it, and by how much. A negative NCU score indicates the model became less confident in the correct answer after seeing the context, a phenomenon the authors term Negative Transfer.
Methodology and Experimental Design
The evaluation covers four models: Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-72B-Instruct, and GPT-4o-mini as a proprietary commercial baseline. The Qwen family allows clean intra-architecture scaling comparisons while holding pre-training philosophy constant. GPT-4o-mini is included as a representative of the closed-API paradigm, with its associated RLHF alignment and likely MoE architecture.
Generation was capped at five tokens with greedy decoding, which is a deliberate constraint to isolate pure extraction behaviour and prevent Chain-of-Thought reasoning from confounding the signal. Three QA domains were evaluated across 1,000 samples each. Adversarial contexts were generated using synthetic fact perturbation. A probability threshold of P > 0.05 was applied for discrete accuracy classification, which is conservative but defensible given vocabulary sizes exceeding 100,000 tokens.
The experimental design is methodologically sound for the specific claim being made. The five-token cap does restrict generalisability, and the authors are upfront about this, but it is the right choice for isolating the extraction mechanism they are trying to measure.
Key Findings
The results are striking and worth examining carefully:
- Oracle accuracy scales inversely with parameters. Qwen 1.5B achieved 95.9% oracle accuracy versus 91.8% for Qwen 72B and 90.1% for GPT-4o-mini. Smaller models followed the context more faithfully when the context was clean.
- Bounded NCU scores show near-parity across open-weight models. The 1.5B and 7B models scored 0.864, the 72B scored 0.868. The difference is negligible. GPT-4o-mini scored 0.777, a meaningful gap.
- Negative Transfer is severe and scale-correlated. Raw NCU scores tell the real story: Qwen 1.5B averaged 0.620, Qwen 7B averaged -74.1, Qwen 72B averaged -20.9, and GPT-4o-mini averaged -7779.3. The extreme magnitude for GPT-4o-mini is partly a mathematical artefact of dividing by near-zero initial entropy, but the underlying confidence collapse is real and systematic.
- Conflict accuracy increases with scale, but this is the wrong metric. GPT-4o-mini "won" the conflict condition with 47.1% accuracy, but this is because it defaulted to its parametric priors rather than following the adversarial context. In a genuine RAG deployment where the retrieved document contains correct but unfamiliar information, this behaviour is a failure mode, not a success.
- Latency differences are substantial. The 1.5B model processed queries in 0.041 seconds per query versus 2.982 seconds for GPT-4o-mini, a roughly 70x difference.
Implications, Limitations, and Open Questions
The authors propose an "Alignment Tax Hypothesis" to explain the commercial model's behaviour: heavy RLHF or constitutional AI constraints may actively resist out-of-distribution updates, causing the model to treat factual conflicts as adversarial inputs to be rejected rather than evidence to be processed. This is a plausible mechanism, but it remains correlational. Without access to GPT-4o-mini's training details, the causal chain cannot be established. The authors acknowledge this directly, and it is the right call to frame it as a hypothesis rather than a finding.
The comparison between open-weight dense models and a proprietary likely-MoE system introduces confounds that cannot be fully controlled. Tokenization differences, pre-training data composition, and unknown alignment procedures all vary simultaneously. The NCU metric normalises for some of these (tokenization length bias), but not all.
The synthetic adversarial construction using the Faker library is a reasonable starting point, but it raises a genuine concern the authors flag: large models may have implicit anomaly detection capabilities that flag synthetic perturbations as implausible, leading them to reject the context not because of prior dominance per se, but because the context looks fabricated. Distinguishing these two failure modes requires adversarial contexts constructed from real-world factual disagreements, which is a harder but more ecologically valid test.
The five-token generation constraint is necessary for the metric's mathematical validity but limits the scope of the claims considerably. Multi-hop reasoning, synthesis tasks, and free-form generation are all excluded. The paper is careful about this, but readers should not generalise the SLM-superiority finding beyond strict extraction workflows.
Despite these constraints, the paper makes a genuine contribution. The NCU metric is a cleaner and more informative signal than EM or F1 for evaluating context utilisation specifically. The finding that SLMs match large models on bounded NCU while offering substantial latency advantages has direct practical implications for RAG system design. The architectural routing hypothesis (Figure 6 in the paper) is a sensible operational takeaway: route extractive tasks to SLMs, reserve high-capacity models for synthesis tasks where parametric depth is genuinely needed.
The most important follow-up study would compare base models against their instruction-tuned counterparts within the same architecture family, which would begin to isolate whether alignment or raw scale is the primary driver of prior dominance. That ablation, combined with evaluation across multiple commercial endpoints, would considerably strengthen the causal claims the current work can only gesture towards.
The full paper is available at https://arxiv.org/abs/2606.23695.