Most benchmarks for medical AI assume that data arrives in a convenient form: structured, regularly sampled, and complete. Real ICU data is none of these things. Vital signs, laboratory values, and intervention records accumulate at wildly different frequencies, with gaps that can span hours. A benchmark that fails to capture this irregularity will systematically overestimate how capable a model actually is when deployed in a clinical setting.
CLIR-Bench is a direct response to this problem. The paper introduces a purpose-built benchmark for question answering over sparse, asynchronous clinical time series drawn from de-identified ICU records. The central question it poses is not just whether a model gets the right answer, but whether it gets the right answer for the right reason, grounded in the correct temporal observations.
What the Benchmark Contains
CLIR-Bench comprises 6,600 multiple-choice QA instances covering 11 clinical variables, including vital signs such as heart rate and laboratory measurements such as lactate. The benchmark is organised around four capability dimensions:
- Temporal understanding: identifying when events occurred and how values changed over time
- Reasoning: drawing inferences from sparse, asynchronous observations
- Forecasting: predicting future clinical states from irregular trajectories
- Decision-making: recommending or evaluating interventions given observed patient context
These dimensions decompose into 11 distinct subtasks. Every QA instance is paired with explicit timestamp-level evidence and a deterministic derivation rule for the correct answer. This evidence-auditable design is the benchmark's most distinctive feature. It allows evaluators to isolate exactly which observations support the correct answer, then systematically perturb or remove that evidence to probe whether models are actually using it.
Construction Pipeline and Methodology
The benchmark was built through a four-stage pipeline: time series extraction from de-identified ICU records, task instantiation encoding analytical goals in executable form, QA pair generation with linked temporal evidence, and human verification. The formal problem is defined cleanly: given a patient trajectory as a set of tuples containing timestamp, clinical variable, measured value, and optional intervention context, a model must answer a natural language question by reasoning over that irregular record.
The multimodal framing here is worth unpacking. Models receive the time series in serialized text form alongside the question and answer options. Four input conditions are evaluated: full time series, question and options only, gold evidence only, and the full time series with gold evidence removed. This factorial design lets the authors diagnose exactly where models fail, whether through inability to retrieve relevant observations, failure to reason once evidence is provided, or reliance on shortcuts that bypass the time series entirely.
Compared to existing time series QA benchmarks such as TimeSeriesExam, Time-MQA, and TSAQA, CLIR-Bench is the only one that explicitly handles irregular sampling, missing data, and all four capability dimensions simultaneously. The comparison table in the paper makes this gap clear.
Results: Models Struggle Badly
The experimental results are sobering. Across 16 models tested, including closed-source systems like GPT-5.4 mini and Gemini-2.5-flash, open-source general LLMs, and specialist time-series LLMs, the best overall accuracy achieved is 50.15% (GPT-5.4 mini). Most open-source models and all time-series specialist models cluster much closer to the 25% random baseline.
Key findings from the evaluation include:
- Time-series specialist models (ChatTS, TimeLLM, ITFormer, AutoTime) perform no better than random on most tasks, suggesting that existing time-series pretraining does not transfer to irregular clinical reasoning
- Performance is highly uneven across subtasks. Models do comparatively better on tasks with strong clinical prior cues (intervention response, summarization) but fail on tasks requiring precise temporal grounding
- Adding the full time series context does not reliably improve accuracy over QA-only inputs for many models, and in some cases actively hurts performance. This suggests models cannot efficiently retrieve relevant evidence from long, sparse records
- Even when provided only the gold supporting evidence, several models still fail, indicating that the reasoning deficit is not purely a retrieval problem
The QA-only versus Full-TS comparison is particularly telling. If a model performs similarly whether or not the time series is present, it is almost certainly relying on clinical priors or statistical regularities in the answer choices rather than patient-specific observations. For safety-critical deployment, this is a fundamental problem.
Limitations and Implications
The benchmark has some limitations worth acknowledging. All tasks are formulated as four-way multiple-choice, which constrains the evaluation to a format that may not reflect the open-ended nature of real clinical queries. The 25% random baseline also means that a model can score 25% with zero clinical understanding, which compresses the effective performance range. Extending to free-form generation with automated or human scoring would strengthen future iterations.
The data source is a single de-identified ICU dataset, which raises questions about generalization across institutions, patient populations, and clinical workflows. ICU data from one hospital system may have systematic patterns in measurement frequency and variable selection that do not transfer elsewhere.
The paper also does not deeply investigate why time-series specialist models underperform. These models were largely developed for regularly sampled industrial or financial data, and their architectural assumptions (fixed-length windows, uniform tokenization of time steps) may be fundamentally mismatched to asynchronous clinical records. That diagnosis would benefit from ablation studies targeting specific architectural components.
Despite these limitations, CLIR-Bench makes a genuine contribution. The evidence-auditable design is methodologically rigorous and provides a template for benchmarks in other high-stakes domains where grounding matters as much as accuracy. The finding that current models cannot reliably retrieve and use sparse temporal evidence is not surprising, but having a principled benchmark to quantify this failure is valuable for directing research effort.
The immediate implication for the field is that neither scaling general LLMs nor fine-tuning on regular time series is sufficient. Progress on irregular clinical time series reasoning likely requires architectural innovations that handle asynchronous, variable-frequency inputs natively, combined with training objectives that explicitly reward evidence faithfulness rather than just answer accuracy.
The benchmark and data are publicly available at huggingface.co/datasets/winall/CLIR-Bench, and the full paper can be read at arxiv.org/abs/2607.09880.