List of Papers Browse by Subject Areas Author List
Abstract
Medical vision–language models (VLMs) have achieved strong performance on chest X-ray (CXR) interpretation, yet they are predominantly trained and evaluated under single-visit or limited pairwise comparison settings, with insufficient support for modeling richer longitudinal trajectories. We introduce MIMIC-MTE (Multi-Time Events), a comprehensive longitudinal dataset featuring multi-visit trajectories (3–10 events). Each event pairs a timestamped CXR and untruncated radiology reports, rich routine clinical tests, and explicit treatment plans. We define time-grounded clinical tasks spanning next-event forecasting (report generation and severity/progression selection), longitudinal question answering over specified time intervals, and treatment plan inference using temporally bracketing outcome evidence.
To establish foundational baselines, we comprehensively benchmark a diverse suite of medical and general-purpose VLMs (e.g., LLaVA-Med, Hulu-Med, MedGemma, and Qwen2.5-VL) under zero-shot and supervised fine-tuning (SFT) settings. Our extensive evaluation reveals that off-the-shelf models exhibit substantial limitations in temporal grounding and multi-event evidence aggregation. Conversely, SFT on MIMIC-MTE significantly unlocks their longitudinal reasoning capabilities, establishing a robust benchmark to drive future research in time-aware clinical foundation models. The data preprocessing pipeline is available on
\href{https://huggingface.co/datasets/wanqi16/MIMIC-MTE}{MIMIC-MTE}.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4415_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://huggingface.co/datasets/wanqi16/MIMIC-MTE
Link to the Dataset(s)
N/A
BibTex
@InProceedings{YanWan_TimeGrounded_MICCAI2026,
author = { Yang, Wanqi AND Chen, Ling AND Zhang, Jianpeng AND Xie, Yutong},
title = { { Time-Grounded Clinical Prediction from Longitudinal Multimodal Patient Histories } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16887},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors compile a multimodal longitudinal dataset by merging existing MIMIC resources (CXR images, ImaGenome labels, EHR records, ECG) into a unified sequence format consumable by vision-language models. They define three generative tasks: (1) prediction of current chest X-ray findings (report, severity, change) from prior patient data, (2) narrative summarization of patient history across modalities, and (3) treatment prediction given history and current state. They evaluate zero-shot VLM baselines and a supervised fine-tuning (SFT) variant, reporting that SFT improves over zero-shot across the proposed tasks and on two adjacent longitudinal CXR benchmarks (MS-CXR-T, Med-Diff-VQA).
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The dataset construction integrates several MIMIC-derived resources (CXR, ImaGenome, EHR, ECG) into a single patient-centric longitudinal sequence format.
The three proposed tasks target distinct aspects of longitudinal reasoning (current-state prediction, narrative summarization, treatment prediction) and, if well-specified, could be useful probes for multimodal VLM capabilities in the clinical setting.
The zero-shot evaluation provides a useful negative result: current general-purpose VLMs do not perform well out of the box on these tasks, motivating targeted development.
SFT on the proposed data improves over zero-shot baselines on the two external benchmarks (MS-CXR-T, Med-Diff-VQA), suggesting the curated data carries signal beyond in-distribution evaluation.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
No methodological contribution and the dataset is a compilation, not a new resource. All underlying annotations already exist and are publicly available. The novelty is limited to a heuristic merging pipeline and a task definition layer derived from existing structured labels, without new clinical annotations, new imaging data, or clinician validation of the derived task labels.
Task framing is misleading. Task 1 is labelled as “next-event” prediction but, as described, predicts the current state (report, severity, change label) given prior context and the current patient data - not a future event.
Missing longitudinal CXR baselines. Prior work on longitudinal CXR modelling is not included in the experiments, so it remains unclear if the provided context and SFT helps in this task over imaging-only variants.
The paper does not provide a qualitative example of an input sequence or an expected output for any of the three tasks. This makes it difficult to assess whether the tasks are well-posed, clinically meaningful, or merely reformulations of existing label prediction problems.
Training details are essentially absent. The paper does not report the optimizer, learning rate, fine-tuning strategy, batch size, epochs, etc.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
to strengthen the paper the authors could consider:
- provide at least one full qualitative sample per task
- rename or reframe Task 1
- include full hyper parameters
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The dataset contribution is a heuristic compilation of existing, publicly available MIMIC-derived annotations, without new imaging data, new clinical labels, or clinician-validated task specifications. Furthermore, it is hard to understand the details of the proposed task formulations in their current presentation.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The rebuttal addresses two of my specific concerns. The authors clarify that Task 1 holds out all fields of ei+1, correctly correcting my earlier reading, and provide concrete SFT hyperparameters.
However, my core concerns remain. The paper is submitted as a methodological contribution, which per the MICCAI submission guidelines should “clearly demonstrate innovations and contributions over state-of-the-art methodologies.” The work proposes no new model, training objective, or architectural component; the modelling reduces to SFT of existing VLMs on curated data, and this bar is not met.
The contribution then rests on the dataset and proposed tasks. The use of templated logic rather than LLM-generated text for the data representations is a positive of the work, and I agree with the authors that the data itself therefore does not require additional clinical verification. However, the proposed tasks are a different matter, and two of the sub-tasks lack clear clinical motivation. The Report Generation sub-task of Task 1 asks the model to generate a comprehensive radiological description for an X-ray held out from the input, which is conditional language modeling on history rather than report generation in any clinical sense. Task 3 (Treatment Inference) predicts an administered treatment while being given the post-treatment image and report of ei+1, which Reviewer 2 noted can openly contain post-treatment markers (“status post X,” etc.). The rebuttal’s response does not address this concern. More clinically grounded alternatives such as outcome prediction or vital-sign forecasting are not proposed.
I agree with Reviewer 3 that there is genuine merit in the proposed dataset and in the direction of multimodal longitudinal benchmarks, and I would encourage the authors to refine and resubmit this work. In its current form, however, without methodological novelty under MIC and without clinically motivated tasks and evaluation that would justify the work as an application contribution, I maintain my recommendation to Reject.
Review #2
- Please describe the contribution of the paper
MIMIC-MTE builds a longitudinal chest X-ray benchmark by linking Chest ImaGenome, MIMIC-CXR-JPG, MIMIC-IV, and MIMIC-IV-ECG into 36,928 patient-disease timelines with 3 to 10 events and 169,412 total events. Each event includes a timestamped CXR, report text, routine clinical tests, and treatment information, and the benchmark defines next-event forecasting, time-bounded longitudinal QA, and treatment inference tasks. Experiments compare several medical and general vision-language models in zero-shot and supervised fine-tuning settings, reporting large gains after fine-tuning on the proposed tasks and improved performance on pairwise temporal benchmarks.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The resource is materially larger and richer than typical pairwise temporal CXR settings, with 36,928 timelines, 169,412 events, 3 to 10 visits per timeline, and linked images, full reports, clinical tests, treatments, and optional ECG context.
- Table 1 gives concrete evidence that the benchmark is genuinely longitudinal rather than a thin pairwise extension: median timeline duration is 103 days, median adjacent gap is 5 days, same-day repeats are common, and 32.29% of timelines contain both improved and worsened states.
- The task suite covers several distinct temporal behaviors instead of only prior-current comparison, including next-event forecasting, interval-bounded aggregation, and treatment-related prediction.
- Benchmarks span multiple open medical and general VLMs in both zero-shot and supervised settings, and the fine-tuned models show large gains across all three task families. For example, Qwen2.5-VL-7B improves from 25.24 to 59.78 Macro-F1 on next-event forecasting, from 38.59 to 65.09 on longitudinal QA, and from 9.60 to 31.61 treatment entity micro-F1. - Targets are derived from existing structured sources rather than LLM-generated labels, and the paper reports missingness and split sizes, which helps readers assess coverage limits.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The data construction is not specified precisely enough for a benchmark paper. The manuscript does not give exact alignment rules for attaching MIMIC-IV labs, vitals, medications, procedures, and ECGs to each imaging event, or explain how same-day repeated studies, conflicting timestamps, and ambiguous matches are handled.
- Split integrity is not fully demonstrated after regrouping records into patient-disease trajectories. Hashing JSON samples does not rule out overlap of underlying patients, studies, images, or events across train, validation, test, or transfer benchmarks, and the text only explicitly states patient separation between training and testing.
- The treatment inference task has a serious leakage risk. Predicting treatment at time t while giving the model the next visit image and report can expose direct mentions of procedures, devices, or post-treatment findings, so the current setup may measure retrospective attribution more than treatment reasoning.
- The empirical study does not test whether the models actually use temporal structure or the added non-image modalities. There are no last-visit-only, shuffled-order, no-timestamp, report-only, image-only, tests-only, or simple sequential baselines.
- Task 1 and Task 2 results are under-specified. Each contains multiple discriminative subtasks, but Table 3 reports a single Macro-F1 per task family even though severity and comparison labels are missing for about 44 to 45% of events. The evaluation subset and missing-label handling are unclear.
- Reproducibility details are missing, including multimodal input serialization for up to 10 visits, context truncation rules, image token budgeting, prompt templates, fine-tuning method and hyperparameters, task sampling during SFT, and the treatment parsing pipeline.
- Some claims are stronger than the evidence supports. The work is mainly a dataset integration and benchmarking contribution on top of public MIMIC resources, while claims about state of the art, external generalization, and clinical decision support are not established by the current comparisons.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Current evaluations are being conducted on self-constructed datasets, which poses a risk of closed-loop evaluation.
To mitigate this, it would be best to incorporate additional external datasets to provide better support/generalizability for the results.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I think authors have addressed most of my major concerns.
Review #3
- Please describe the contribution of the paper
The primary contribution of this manuscript is the introduction of MIMIC-MTE (Multi-Time Events), a comprehensive, longitudinal multimodal dataset designed to capture multi-visit clinical trajectories consisting of 3 to 10 events. Additionally, the authors define three time-grounded clinical tasks to evaluate longitudinal reasoning: next-event forecasting, longitudinal question answering, and interventional treatment inference. Finally, the paper establishes strong foundational baselines by benchmarking and fine-tuning several state-of-the-art vision-language models (VLMs) on this dataset
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Highly Relevant Problem Formulation: The authors accurately identify and address a critical limitation in current medical VLMs, which are predominantly evaluated on single-visit inputs that fail to capture the non-monotonic nature of chest and lung pathologies.
Multimodal Richness: Each event within the dataset couples chest radiographs with untruncated radiology reports, routine clinical tests, and explicit interventional treatment plans, offering a highly realistic clinical snapshot
Comprehensive Benchmarking: The study rigorously evaluates diverse state-of-the-art VLMs under zero-shot and supervised fine-tuning (SFT) settings. The experiments successfully demonstrate that SFT on this specific dataset unlocks crucial multi-visit reasoning capabilities that off-the-shelf models lack.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Limited Data Diversity: The MIMIC-MTE dataset is constructed entirely from MIMIC resources (MIMIC-CXR, MIMIC-IV, and Chest ImaGenome), which restricts the data to a single institution. This limits the proven generalizability of the trained models across diverse hospital systems or demographic populations, a limitation the authors acknowledge as future work.
Lack of Architectural Novelty: The methodology relies entirely on the supervised fine-tuning of existing, open-weight VLM backbones (such as Qwen2.5-VL and MedGemma). The paper does not propose any novel network architectures or attention mechanisms explicitly designed to handle the complexities of irregular temporal gaps or non-monotonic multi-event integration.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
For future revisions, it would be beneficial to include an ablation study or discussion on how token context limits affect the model’s ability to process up to 10 fully multimodal events simultaneously, and whether information retrieval degrades at the tail end of long patient histories.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper deserves an “Accept” due to the high utility and quality of the newly introduced MIMIC-MTE dataset. By moving the field away from limited pairwise comparisons and toward 3-10 event longitudinal trajectories, the authors provide a vital resource for training models that reflect realistic clinical decision-making. Furthermore, the robust experimental design proves that explicit instruction tuning on multi-visit data effectively resolves the catastrophic failure of base models in temporal grounding tasks. While the work is more dataset-centric than architecturally novel, the empirical findings and the public release of the dataset and codebase will significantly drive future research.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal successfully addresses the major concerns raised in the reviews. In particular, the authors clearly clarified that the proposed benchmark is not merely a collection of existing resources, but a structured longitudinal multimodal temporal dataset specifically designed for patient-disease trajectory modeling across multiple clinical events. The distinction from prior single-visit or pairwise longitudinal CXR datasets is now much clearer.
The protocol clarifications were also convincing. The explanations regarding true next-event forecasting, patient-disjoint splits, temporal ordering, alignment procedures, and avoidance of treatment leakage substantially improve confidence in the experimental setup and dataset construction. The rebuttal adequately resolves concerns about ambiguity in event matching and train/test contamination.
The additional discussion of ablation studies is especially valuable. The comparisons between full multimodal temporal input, last-visit-only, image-only, shuffled-order, and no-timestamp settings demonstrate that the benchmark meaningfully evaluates temporal reasoning and multimodal integration rather than simple shortcut learning. The reported degradations on longitudinal QA and treatment modeling tasks strongly support the paper’s claims regarding the importance of longitudinal context.
The reproducibility clarifications are also sufficient for a benchmark/data paper. The authors provided important implementation details, clarified preprocessing choices, and committed to code release without relying on promises of new experiments.
Overall, despite limitations acknowledged by the authors (single-institution source and reliance on existing public datasets), the work presents a valuable and timely benchmark resource for multimodal longitudinal medical AI research. The scale, task diversity, and systematic evaluation make the contribution significant for the MICCAI community.
Author Feedback
We thank the reviewers for recognizing MIMIC-MTE’s clinical motivation, longitudinal multimodal scale, task breadth, and comprehensive zero-shot/SFT benchmarking.
1.Contribution/scope (R1Q1,R2Q7,R3). We respectfully disagree that MIMIC-MTE is merely a compilation. It fills a gap not covered by single-visit or pairwise CXR resources: chest disease courses are irregular, long-horizon, and non-monotonic. MIMIC-MTE converts asynchronous CXR, report, EHR, ECG, and treatment records into patient-disease-specific multi-event visual-language-clinical trajectories, supporting next-event forecasting, interval QA, and treatment inference. Imaging acquisition is derived from public resources, so clinician validation is not required as for newly created subjective labels. Although single-institution, MIMIC-MTE links to MIMIC-IV demographics, which uses six race/ethnicity groups. Multi-institutional expansion and new temporal architectures are future work.
2.Protocol clarifications. (R1Q2) As stated in Sec. 2.2, Task 1 is true next-event forecasting: given only {e1,…,ei}, all fields of ei+1 are hidden and used only as targets. (R2Q1) Sec.2.1 states alignment by a composite key of patient identity, disease category, event identifier, and timestamp. event ID=subject_id+incremental code. Same-day studies have fine timestamps, are kept separate and time-ordered; no unresolved conflicts/ambiguous matches remain. (R2Q2) Splits are patient-disjoint; events/images are nested by patient, and train/test patient/event/image overlap is 0.(R2Q3) Treatment leakage. Task 3 does not provide target treatment records. ei/ei+1 provide only CXR+disease-state description, not treatment notes or MIMIC-IV med/procedure labels. (R2Q5) Macro-F1 uses only non-missing severity/comparison labels; Table3 averages discriminative subtasks, and final will report per-subtask scores/counts.
3.Baselines (R1Q3,R2Q4). We performed controlled Qwen2.5-VL ablations under both zero-shot and SFT: last-visit-only, image-only, report-only, shuffled-order, and no-timestamp. These variants directly test the roles of multi-event history, temporal grounding, and non-image context. Last-visit-only barely affects PNC, but strongly degrades LQA, e.g., SFT BL-avg 8.65->5.89 and Macro-F1 65.09->36.38, showing that interval QA requires multi-visit evidence rather than the latest event alone. PTM also drops in zero-shot entity-F1 from 9.60 to 4.76.Temporal cues are important: shuffled-order causes modest drops, while removing timestamps severely hurts time-bounded LQA, especially SFT Macro-F1 65.09->44.15.Modality ablations show complementary roles. Report-only remains close to full input, indicating that radiology text carries strong disease-state signal. In contrast, image-only substantially hurts generative and treatment outputs: PNC zero-shot RadGraph-F1 drops 11.22->4.95, PNC-SFT ROUGE-L-F1 drops 22.11->19.71, LQA-SFT ROUGE-L-F1 drops 21.02->18.01, and PTM zero-shot entity-F1 drops 9.60->2.01.However, image-only remains close on discriminative severity/progression, e.g., PNC-SFT Macro-F1 59.78->58.67 and LQA-SFT 65.09->64.93.Thus, CXR-only history can support coarse state classification, but multimodal longitudinal context is needed for report generation, interval reasoning, and treatment inference. Prior longitudinal CXR works are mainly pairwise image classifiers and cannot perform our full task suite; image-only/last-visit-only are the fairest in-domain baselines.
4.Reproducibility/examples (R1Q4-5,R2Q6). The demo video in the appendix contains input/output examples; we will add more cases. We do not truncate context. Images are original event-linked MIMIC-CXR-JPG with official VLM preprocessing, no extra dropping/pruning. Qwen SFT: AdamW, LR1e-4, WD0.1, cosine+3% warmup, 1 epoch, batch16, LoRA r32/alpha64/drop0.05, image range 256x28x28-768x28x28; other models use defaults. We will release the full code for more details, e.g., prompt, parsing code, and treatment details.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Reviewers appreciated the clinical relevance, scale, multimodal richness, and the move beyond pairwise CXR comparison toward multi-event histories.
However, the reviews are divided: while one reviewer strongly supports the dataset contribution, others question the novelty of compiling existing public resources and raise substantial concerns about task clarity, data-alignment rules, potential treatment leakage, missing temporal/modality ablations, and incomplete reproducibility details. Overall, the paper has potential as a useful benchmark resource, but acceptance would require a convincing rebuttal clarifying the task definitions, split integrity, data construction pipeline, leakage controls, and fine-tuning/evaluation protocol.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper presents a timely and useful longitudinal multimodal benchmark built from MIMIC-derived resources, with tasks designed to test multi-event clinical reasoning. Although the work has limited methodological novelty and some tasks should be more carefully framed clinically, the rebuttal clarifies settings and ablations. Given the scale of the resource, the value of the benchmark, and the generally positive post-rebuttal assessment, I recommend acceptance, with the final version clearly discussing limitations around single-institution data and clinical task validity.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
This manuscript received mixed post-rebuttal reactions. Concerns over data leakage, missing protocol details, comparisons with baselines, etc. were addressed in the rebuttal, while concerns over the limited methodological contribution remain. As noted by R1, while the research question is of merit, the paper suffers from unclear technical novelty (focusing on dataset linkage and reorganization instead) when positioned as a methodology paper, and some of the proposed tasks suffer from unclear real-world clinical relevance.