Language models are increasingly being positioned as reasoning engines over clinical data, yet a fundamental architectural problem has gone largely unexamined in medical contexts: the tendency of transformer models to retrieve information from the centre of a long input context far less reliably than from the edges. A new paper from arXiv (arXiv:2608.20348) provides the first systematic empirical characterisation of this phenomenon in real electronic health record settings, quantifies its clinical stakes, and proposes a lightweight mitigation that outperforms several well-established retrieval methods on an end-to-end instruction-following task.
The stakes here are not abstract. A single patient record from an integrated delivery network can exceed 100,000 tokens. Critical clinical facts, a discontinued contraindicated medication, a lab value flagging renal impairment, an old diagnosis that invalidates a current treatment plan, can sit anywhere in that timeline. If a model is systematically less accurate at the 60th percentile of a document than at the 5th, that is not a benchmark quirk. It is a patient safety issue.
What the Paper Contributes
The authors make four distinct contributions, which are worth separating clearly because they address different parts of the problem:
- Empirical characterisation of the clinical lost-in-the-middle (CLitM) effect. Using MedAlign, a dataset of 983 clinician-authored instruction-response pairs over real patient EHRs, they run needle-in-a-haystack experiments across six language models, producing 2,196 model-by-instruction observations. The peak-to-trough accuracy gap is 21.9 percentage points (59.5% at the 20-30% position decile versus 37.6% at 70-80%), with clustered confidence intervals that account for the repeated-measures structure across models sharing the same instructions.
- Differential Transformer evaluation on structured EHR prediction. On EHRSHOT laboratory abnormality prediction tasks, Differential Transformer (DiffAttn) shows consistent gains on class-imbalanced tasks (+6.1 pp AUROC on anaemia, +4.2 pp on hyperkalaemia) but degrades on balanced tasks. A per-token QCCS-gated variant collapses under gradient starvation, with gate training converging to chance on hyperkalaemia.
- Query-Conditioned Clinical Suppression (QCCS). A lightweight character n-gram gate trained on MedAlign instruction-response pairs using lexical overlap labels. The gate scores EHR sentences for relevance to the clinical query and selects a compact context for downstream inference.
- A multi-arm end-to-end evaluation comparing QCCS against BM25, section-header-filtered BM25, dense retrieval, and cross-encoder reranking on a held-out test split of 83 instructions, with LLM-as-judge semantic scoring validated by a second independent judge (Cohen's kappa = 0.767 on the QCCS arm).
The Counterintuitive Finding on Retrieval Recall
The most striking result is not that QCCS works, but why it works despite having dramatically lower retrieval recall than every comparator. At k=20, BM25 retrieves the gold-standard evidence sentence in 98.8% of test instructions. Cross-encoder reranking achieves 96.4%. QCCS retrieves it in only 34.9%. By conventional RAG evaluation logic, QCCS should be the worst-performing arm.
It is instead the best, by a large margin. Overall accuracy: QCCS 25.3% [95% CI: 17, 35] versus BM25 2.4%, dense 2.4%, cross-encoder 1.2%, full context 3.6%. The conditional accuracy analysis is the key diagnostic: BM25 achieves only 2.6% Stage-2 accuracy even in the 78 of 79 instructions where it successfully retrieves the gold evidence sentence. QCCS achieves 25.0% accuracy in the 52 instructions where it does not retrieve the gold sentence at all.
This dissociation between retrieval recall and downstream accuracy is the paper's central empirical claim, and it deserves scrutiny. The authors' interpretation is that query-aligned context selection reduces distractor interference, preventing the reader model from confabulating plausible but incorrect answers anchored on retrieved but irrelevant sentences. The qualitative failure analysis supports this: 64.2% of BM25 failures are classified as wrong-entity confabulation, where the model executes the correct reasoning type but anchors on a distractor. Only 1.2% are outright refusals.
The implication is that for EHR instruction-following, the bottleneck is not evidence retrieval but evidence isolation. A context containing the correct sentence alongside 19 distractors may be harder for a 7B-parameter reader to reason over than a context containing no perfect match but high query coherence.
Methodological Considerations
Several aspects of the experimental design warrant careful reading. The test split is small: 83 instructions across 74 patients. The authors are transparent about this, framing the work explicitly as a proof-of-concept pilot. The confidence intervals are correspondingly wide, particularly for position-stratified analyses where some bands contain as few as 9 instructions. The 16.7% middle-position accuracy for QCCS carries a 95% CI of [3, 30], which spans nearly the full range of plausible outcomes.
The LLM-as-judge evaluation methodology is reasonable given the absence of structured ground-truth labels for free-text clinical responses, and the second-judge validation (Claude Sonnet 4.6, kappa = 0.767) provides meaningful inter-rater reliability evidence. The token-overlap secondary metric is included and shown to be misleading, particularly for cross-encoder responses, where it inflates accuracy from 1.2% to 9.6% overall by rewarding vocabulary matching without correct answers. This is a useful methodological caution for the broader RAG evaluation literature.
The QCCS gate architecture robustness analysis across 27 variants (n-gram order, embedding dimension, MLP depth) is thorough, with performance varying within 6.25 pp across the full sweep. The production configuration (n-gram=3, embed_dim=64, standard MLP) is not obviously special, which is reassuring. The no-query ablation in Appendix H, showing that query conditioning accounts for +23.8 percentage points of middle-position recall, is the clearest evidence that the gate's benefit is specifically attributable to query alignment rather than generic context compression.
The QCCS-DiffAttn per-token gating result is an honest negative. Gate training collapses to chance on hyperkalaemia and underperforms scalar DiffAttn on all four EHRSHOT tasks. The focal BCE ablation partially recovers gate non-degeneracy but does not recover AUROC. This is a meaningful null result: integrating a learned selection gate directly into the attention mechanism as a per-token inhibition weight does not appear to transfer gains from the context selection setting to the structured prediction setting, at least under the training conditions evaluated here.
Limitations and Open Questions
The authors are candid about what the paper does not establish. No clinician evaluation was conducted. The 25.3% absolute accuracy achieved by QCCS is insufficient for autonomous clinical decision support; the paper explicitly states that human-in-the-loop oversight is required and that prospective validation with practicing clinicians is a prerequisite for deployment. The reader model, Qwen2.5-7B-Instruct with a 16k-token context window, is chosen for accessibility; the 14B matched experiment in Appendix P confirms the directional pattern but absolute numbers remain low.
Several questions remain open. The CLitM characterisation uses six MedAlign-released models, none of which are current frontier models. Whether the 21.9 pp positional gap persists in GPT-4-class or Llama 3.1 70B models is unknown. The MedAlign dataset, while clinician-authored, covers a specific patient population and EHR system; generalisation to other health systems, note styles, or clinical domains is unvalidated. The QCCS gate is trained on lexical overlap labels, which may not capture semantic relevance for paraphrased or abstracted clinical facts.
The broader question the paper raises is whether the RAG paradigm, as conventionally implemented for clinical applications, is solving the right problem. If retrieval recall is near-perfect but downstream accuracy remains near zero, the failure mode is in context presentation and reader reasoning, not in document retrieval. That reframing has practical consequences for how clinical NLP systems should be architected and evaluated.
Code is available at github.com/sanjaybasu/inhibitory-attention-ehr. The full paper is at arXiv:2608.20348.