Predictive models for ICU mortality have reached impressive discrimination, but accurate prediction and clinically actionable explanation are not the same thing. A SHAP waterfall chart telling a clinician that blood urea nitrogen ranked third in feature importance does not, by itself, constitute a bedside explanation. This paper, available at arXiv:2608.26109, takes a direct run at that gap by asking whether a large language model, either standalone or embedded in a structured agentic pipeline, can convert model outputs into clinically meaningful narratives. It is a feasibility study in the honest sense: small dataset, explicit uncertainty, and appropriately hedged conclusions.
What the Paper Contributes
The core comparison is between two explanation architectures applied to the same XGBoost mortality predictions. The first is a standalone LLM prompt that receives patient data and SHAP attributions and returns a single narrative. The second is a four-step agentic pipeline that separates data interpretation, guideline checking, differential construction, and final synthesis into discrete stages. The motivation for decomposition is reasonable: critical care reasoning genuinely does involve distinct cognitive operations, and collapsing them into one prompt risks producing fluent but poorly grounded output.
Beyond the architectural comparison, the paper makes a secondary contribution by being unusually transparent about data quality. Missing value rates for clinically critical variables are reported explicitly: mean arterial pressure was absent in 84.7% of rows after physiologically implausible values were excluded, temperature in 93.4% after a Fahrenheit outlier was harmonised, and lactate in 81.5%. Surfacing these figures matters because an LLM fed sparse data can still produce confident-sounding prose, which is precisely the kind of failure mode that makes AI explanation dangerous in high-stakes settings.
Methodology
The dataset is the eICU Demo cohort, 2,353 ICU stays with 8.1% mortality, which is small but publicly accessible and therefore auditable. XGBoost achieved an AUROC of 0.855 (95% CI 0.796 to 0.906) and an AUPRC of 0.332 (95% CI 0.217 to 0.494) on the held-out test set of 471 encounters, with confidence intervals derived from bootstrap resampling. The AUPRC figure deserves attention: at 8.1% prevalence, a value of 0.332 represents meaningful lift over the no-skill baseline of roughly 0.081, but it also signals that positive predictive value will be limited in practice.
Explanations were generated for a stratified 38-case subset. Evaluation metrics included SHAP alignment measured by Jaccard similarity between narrative factors and top SHAP features, direction consistency, guideline grounding, value specificity, and plausibility. Leakage, defined as explicit mention of the predicted outcome in the explanation, was also tracked. Only 14 of the 38 cases overlapped with the retained SHAP review subset, which is the primary constraint on statistical confidence throughout.
Key Findings
- The standalone LLM produced one explanation with explicit outcome leakage; the agentic pipeline produced none across the 38-case subset.
- SHAP alignment favoured the standalone LLM substantially: mean Jaccard 0.171 versus 0.077 for the agentic pipeline, with direction consistency at 92.9% versus 78.6%.
- The agentic pipeline scored higher on guideline grounding (0.762 versus 0.143), value specificity (0.236 versus 0.143), and plausibility (0.700 versus 0.671).
- Top predictive features from XGBoost included age, oxygenation indices, blood urea nitrogen, lactate, and respiratory rate, all physiologically coherent markers of instability.
The tension between SHAP alignment and guideline grounding is the most analytically interesting result. The agentic pipeline produced explanations that were more grounded in formal clinical criteria and more specific about patient values, but it did so at the cost of fidelity to the features that actually drove the model. This is not a trivial tradeoff. A clinician reading an agentic explanation that emphasises sepsis criteria might reasonably assume that sepsis criteria drove the risk score, when in fact the dominant model signal came from a different feature. That kind of misalignment could be more dangerous than a vague explanation, because it has the surface appearance of rigour.
Limitations and Broader Implications
The authors are candid about the study's constraints. The eICU Demo dataset is far smaller than full-scale ICU cohorts used in comparable predictive modelling work. The 14-case overlap between explanation and SHAP review subsets means that confidence intervals around the comparative metrics are wide enough to preclude strong conclusions. Both generators used a single local base model, so the observed differences reflect one specific implementation rather than any general architectural truth. The attribution comparison also depends on heuristic mapping from free-text narrative factors to structured model features, which introduces its own noise.
The automated metrics are proxies, not substitutes for expert clinical evaluation. Plausibility scores and guideline grounding counts can be gamed by a model that knows what clinicians want to read, and neither metric captures whether the explanation would actually change a clinician's decision in a useful direction. Prospective evaluation with critical care physicians remains the necessary next step, and the paper appropriately frames this work as a feasibility study rather than a validation.
For the field, the paper's most useful contribution may be methodological rather than empirical. It operationalises the distinction between model fidelity and clinical grounding as competing objectives, and it shows that optimising for one can reduce the other. That framing suggests that production clinical explanation systems probably need both components running in parallel: an attribution layer to track what the model actually did, and a language layer to translate that into readable clinical narrative, with explicit reconciliation between the two. Neither component alone appears sufficient.
The data quality transparency also sets a useful norm. Missing value rates of 80 to 93 percent for variables like MAP, temperature, and lactate are not unusual in real ICU data, but they are rarely foregrounded in papers that then proceed to generate LLM explanations as if the inputs were complete. An explanation system that silently imputes or ignores missing values while producing confident prose is a patient safety concern, and calling that out explicitly is the right approach.
Full paper: arXiv:2608.26109