List of Papers Browse by Subject Areas Author List
Abstract
We introduce LongMedBench, a real-world EHR benchmark for long-horizon clinical reasoning with medical agents. Prior evaluations of LLM-based medical agents have largely emphasized short-context medical QA, tool use, or single-encounter interactions, whereas real-world care requires integrating evidence across repeated visits, tests, treatments, and discharge outcomes. LongMedBench integrates MIMIC-IV admission records and clinical notes into time-series event streams and controlled long-context memory datasets. It comprises 355 patients, averaging 19.72 inpatient visits per patient and 44.91 medical events per visit. We propose an evaluation taxonomy with three suites: factual QA, temporal reasoning, and long-horizon decision-making. These suites measure how agents retrieve, order, and use historical patient information over extended horizons. Experiments show that recent LLMs use explicit timestamps effectively but still struggle with implicit temporal inference. Longer context, RAG, and agent memory improve factual retrieval more than long-horizon decision-making, revealing a gap between accessing historical evidence and using it for clinical decision reasoning.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4292_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Yanzhen-CN/LongMedBench
Link to the Dataset(s)
https://physionet.org/content/mimiciv/3.1/
BibTex
@InProceedings{CheYan_LongMedBench_MICCAI2026,
author = { Chen, Yanzhen AND Xu, Zihan AND Zhang, Xiaocheng AND Fan, Zhiting AND Zhai, Weiqi AND Xu, Hongxia AND Liu, Zuozhu},
title = { { LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper starts with constructing a dataset architecture and a taxonomy for studying the medical agents’ temporal sensitivity. The paper then runs experiments to evaluate medical agents in long-horizon clinical reasoning. The paper finds that LLM agents struggle with implicit temporal reasoning for visit-level understanding for patients.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper studies an interesting problem. Understanding the medical LLM agents is important.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper is not very well written, there are many things undefined and not well explained. Moreover, the contribution is incremental. The figures are not very clear. Specifically, the example in figure 2 is not well explained and it does not add understanding to the reader. Moreover, the work is not reproducible. I state here some minor comments on improvements, typos and errors:
- Do not capitalize after the semicolon
- The bold text is excessive
- On page 2, paradigms is going out of paper margin
- Add the article “the” when mentioning the agent
- Add a space between the word and the references
- Explain what a medical event is
- The patient record S, should this have some superscript to indicate which patient?
- On page 4, “an action” not “a action”
- On page 5, “No external tools are” not “is”
- On page 6, “We introduce Mem0” but isn’t Mem0 coming from reference [1]? Therefore, you are not introducing it.
- Also about Mem0, please add an explanation to what it is.
- The references should be capitalized correctly.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
As stated in the weaknesses section, the paper’s contribution is minor and it needs to be better written for easier understanding of the reader.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The authors respond to my comments, but I do not think the paper is ready for publishing at this stage.
Review #2
- Please describe the contribution of the paper
This paper introduces LongMedBench, a MIMIC-IV-based benchmark for evaluating medical agents on longitudinal clinical reasoning across many visits. It organizes evaluation into three task groups: factual retrieval, temporal reasoning, and long-horizon decision-making. The contribution is relevant, but it should be framed as a more temporally focused benchmark design, rather than a wholly new direction, given prior work such as MedAgentBench (Jiang et al., 2025), AgentClinic (Schmidgall et al., 2024), TIMER (Cui et al., 2025), ER-Reason (Mehandru et al., 2025), and EHRSQL (Lee et al., 2022).
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper targets an important problem: real clinical reasoning is longitudinal, and most current medical LLM evaluations still under-test this setting. The benchmark design is reasonably well structured, especially the split between note memory, event memory, and contextual memory, and the temporal reasoning tasks are the strongest part of the paper. In particular, the benchmark clearly separates explicit timestamp use from implicit temporal inference, and the results show that models perform much better on the former than the latter. That is a useful and believable finding. The negative result is also valuable: memory or retrieval helps factual lookup more than actual downstream decision-making.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main weakness is that the paper overclaims novelty. Several recent benchmarks already move in this direction, including MedAgentBench (Jiang et al., 2025), AgentClinic (Schmidgall et al., 2024), TIMER (Cui et al., 2025), and ER-Reason (Mehandru et al., 2025). So the novelty is more in the specific task packaging than in the overall idea. A second issue is that the “long-horizon decision-making” task may not truly measure optimal clinical reasoning, because predicting the next recorded EHR action is closer to trajectory continuation than gold-standard decision evaluation. Third, the paper’s own ablations show that adding more historical memory gives limited benefit, which weakens the claim that the benchmark already isolates a strong long-horizon reasoning bottleneck. Finally, there are some inconsistencies that should be fixed, such as 335 vs. 355 patients and the stated 128K context limit versus 400K for GPT-5-mini in Table 4.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please refer to my previous comments.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal clarifies several concerns. However, I still feel that the decision-making setup is not fully convincing, since the ground truth comes from recorded EHR actions and may not reflect the best clinical decision. I am also not fully convinced that the experiments demonstrate strong use of long-term history, given the limited gains from memory augmentation.
Review #3
- Please describe the contribution of the paper
The paper introduces LongMedBench, a benchmark for evaluating medical agents on long-horizon clinical decision-making using longitudinal EHR data from MIMIC-IV. It constructs dense multi-visit patient trajectories and proposes a hierarchical evaluation framework including factual QA, temporal reasoning, and long-horizon decision tasks. The benchmark aims to assess how models utilize long-term clinical history for reasoning and planning.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths 1.The paper clearly identifies the limitation of existing medical agent benchmarks focusing on short-context QA and tool use, and motivates long-horizon clinical reasoning, which is meaningful and important. 2.It constructs a longitudinal EHR-based benchmark from MIMIC-IV with dense multi-visit trajectories, enabling evaluation beyond single encounters. 3.The hierarchical task design (factual QA → temporal reasoning → long-horizon decision) provides a structured way to evaluate different reasoning levels.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Weaknesses
Benchmark details 1.The data filtering strategy keeps only abnormal lab results and patients with ≥15 hospitalizations. This may introduce strong observation bias, overrepresenting high-acuity patients and removing normal clinical signals. 2.In Sec. 2.2 (Long-Horizon Decision Making), the ground truth is sampled from historical actions. However, clinical decision-making often allows multiple valid next steps. The current formulation still assumes a single correct answer, even with the 24h time-decay mechanism. It is unclear how clinical correctness is ensured under this setup. 3.There is an inconsistency in the reported dataset size. The abstract mentions 335 patients, while Fig. 1 and Sec. 2.1 report 355 patients.
Experiment 1.Temporal reasoning performance is strong with explicit timestamps but drops significantly without them. Does this indicate that models rely heavily on explicit temporal markers rather than true implicit temporal reasoning, especially in visit-level and joint sorting tasks? 2.Increasing history length or introducing RAG and Mem0 brings limited improvement in long-horizon decision tasks. Does this suggest that the benchmark does not strongly depend on long-term memory as intended, and that performance is mainly driven by immediate context? 3.Factual QA scoring relies on LLM-based matching and keyword coverage. How is the reliability of this evaluation validated, for example agreement with human annotations or robustness to paraphrasing?
Presentation 1.There are typos in Fig.2, “Cheif Complaint” should be “Chief Complaint”. 2.Figures are hard to read. The font size is too small, especially in Fig. 2, and the figure is overloaded with information. 3.Some figure content is redundant with the method section, and repeating equations in the figures is unnecessary and affects readability.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important problem and proposes a benchmark for long-horizon clinical reasoning with a clear task design. Overall, it is important, but there are still some concerns. The data filtering may introduce bias, and the decision tasks rely on historical actions as ground truth, which may not reflect real clinical settings.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Thank you for the reply. I appreciate the clarification of the benchmark design and the positioning of the long-horizon decision task. The paper addresses an important problem in long-horizon clinical reasoning with a clear and structured benchmark design. The remaining issues (e.g., factual QA evaluation reliability and potential cohort bias) could be further improved in future work. Overall, I find the response reasonable and will support acceptance.
Author Feedback
We thank the reviewers and the meta-reviewer for their careful feedback. We address the four concerns below and will revise the paper accordingly.
1.Data Bias (abnormal labs, ≥15 visits) (R4, Meta Q1)
(a) Lab filtering follows MIMIC-CDM (Hager et al., 2024) from Nature Medicine, which showed that normal ranges act as distractors in longitudinal reasoning. We agree this creates a high-acuity cohort and will add a scope statement in Sec. 2.1.1.However, this makes the evaluation more clinically realistic. Like real-world clinicians, the agent should reason from salient signals, not passively absorb all measurements.
(b) ≥15 visits balances sufficient length for constructing long-horizon tasks and ensuring an adequate patient count. At 20 visits, only ~100 patients remain; at 10, visit-level sorting/ranking tasks become infeasible. At 15, we retain 355 patients, comparable to existing benchmarks: 276 for MedAlign (Fleming et al., 2024), and 511 for DiReCT (Wang et al., 2024). We acknowledge this scope limitation and will discuss this trade-off in the revision.
2.Experiment Questions (R4, R2, Meta Q2)
We respectfully argue that the majority of R4’s comments on the experiments are empirical findings, not design flaws. They reveal model limitations in implicit temporal reasoning, memory mechanisms, and history integration.
(a) Implicit temporal reasoning (R4). The τ drop without timestamps reveals that current LLMs have not internalized temporal structure from clinical narratives, a deficit invisible to prior temporal QA benchmarks. Our paired design intentionally isolates this bottleneck.
(b) Marginal benefit from RAG/Mem0/history (R2, R4). The negative result indicates that models can retrieve but cannot integrate cross-visit evidence into decisions. Factual QA degrades with history; decision-making stays flat regardless of memory. Together, they show retrieval is history-dependent while reasoning is not.
(c) Evaluation reliability (R4). For factual QA, to measure precise retrieval, models are prompted with few-shot examples to return the exact ground-truth values without paraphrasing. Answers are evaluated by exact keyword matching following works in general domain like RULER (Hsieh et al., 2024) from COLM 2024.We will add the datail to the task settings.
(d) Decision making (R2). We agree that the next recorded EHR action is not a gold-standard decision. This should be interpreted as whether an agent can combine immediate context with long-horizon history to recover a trajectory-consistent, clinically plausible near-future action. The recorded action is thus an observational reference. Our 24h decay provides tolerance for temporally adjacent actions, but we will explicitly state that it only partially addresses the existence of multiple valid next steps.
3.Contribution & Novelty (R1, R2; Meta Q3)
We agree with the partial overlap with recent work. However, our claim is that, LongMedBench is not the first realistic longitudinal medical benchmark in general, but to our knowledge, the first benchmark that jointly evaluates agent-oriented clinical decisions on cross-visit real-world EHR trajectories under controlled note, event, and context memory settings. This setting is distinct from MedAgentBench’s (Jiang et al., 2025) EHR tool-use tasks, AgentClinic’s (Schmidgall et al., 2024) single-encounter diagnostic simulations, TIMER’s (Cui et al., 2025) temporally grounded instruction evaluation, and ER-Reason’s (Mehandru et al., 2025) ER-workflow belief-updating benchmark. We will fix the paper’s novelty claim to this combination.
4.Presentation & Reproducibility (R1, R2, R4, Meta Q4)
Presentation issues will be corrected: the patient count unified to 355, typos fixed, the Mem0 explanation added, and Fig. 2 redesigned with larger fonts and adequate information. We will release dataset construction scripts and evaluation code upon acceptance.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The reviewers appreciated the motivation/importance of the study, the task formulation (which is very timely), and the findings discovered by this curated dataset. As the meta-reviewer of this paper I personally like this paper, but I still wanna invite the authors for a rebuttal to address some potential small concerns raised by reviewers: 1.The potentially introduced data bias which keeps only abnormal lab results and patients with ≥15 hospitalizations 2.Experiments questions raised by R4 (they are very meaningful questions) 3.The differentiation to other related benchmarks (R2) 4.Some presentation problems and typos raised by R1 and R4
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The rebuttal is valid and the motivation of this work shall be appreciated. I give a pass to this paper at this stage but the authors should not consider it as the final product of the work. Some major concerns such as the factual QA evaluation reliability and the human groundtruth annotation shall be considered if the work wanna publish at a higher platform. Overall, I appreciate the authors’ efforts and look forward to seeing the future work of this piece can really contribute to the community.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The majority of reviewers recommend acceptation of the paper. Key strengths are the important problem and proposed LLM to address them. The issues raised by the reviewers are clarified.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The benchmark addresses an important need for evaluating long-horizon clinical reasoning and decision-making by medical agents, and two reviewers support acceptance after rebuttal. However, one reviewer explicitly maintains a Reject, stating that the paper is not ready for publication, while other comments still question whether recorded EHR actions are an ideal ground truth and whether the experiments fully demonstrate long-term history use.