List of Papers Browse by Subject Areas Author List
Abstract
The rapid progress of multimodal large language models (MLLMs) has led to increasing interest in agent-based systems. While most prior work in medical imaging concentrates on automating routine
clinical workflows, we study an underexplored yet clinically significant setting: distinguishing visually hard-to-separate diseases in a zero-shot setting. We benchmark representative agents on two imaging-only proxy diagnostic tasks, (1) melanoma vs. atypical nevus and (2) pulmonary edema vs. pneumonia, where visual features are highly confounded despite substantial differences in clinical management. We introduce a multi-agent framework based on contrastive adjudication. Experimental results show improved diagnostic performance (an 11-percentage-point gain in accuracy on dermoscopy data) and reduced unsupported claims on qualitative samples, although overall performance remains insufficient
for clinical deployment. We acknowledge the inherent uncertainty in human annotations, conservative binary setting, and the absence of clinical context, which further limit the translation to real-world settings.
Within this controlled setting, this pilot study provides preliminary insights into zero-shot agent performance in visually confounded scenarios. Our code is available at https://github.com/TruhnLab/Contrastive-
Agent-Reasoning.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1616_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/TruhnLab/Contrastive-Agent-Reasoning
Link to the Dataset(s)
derm7pt dataset: https://www.kaggle.com/datasets/menakamohanakumar/derm7pt
MIMIC-CXR-JPG dataset: https://www.physionet.org/content/mimic-cxr-jpg/2.1.0/
BibTex
@InProceedings{ZhaZih_Can_MICCAI2026,
author = { Zhao, Zihao AND Hauke, Frederik AND De Castilhos, Juliana AND Nebelung, Sven AND Truhn, Daniel},
title = { { Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors identify the under-explored yet clinically significant problem of distinguishing visually hard-to-separate diseases in a zero-shot setting, propose the novel contrastive multi-agent framework CARE without extra training, conduct extensive benchmarking on two medical imaging tasks to validate that CARE effectively boosts diagnostic performance and reduces unsupported claims, and offer preliminary insights into zero-shot agent performance for visually confounded medical cases.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors address a clinically meaningful and under-investigated task: zero-shot discrimination of visually confusable diseases. They propose a simple yet effective CARE framework that requires no additional training or fine-tuning. Extensive benchmarking and well-designed ablation experiments are conducted on two representative medical imaging tasks, demonstrating clear performance gains over baseline methods. The framework also helps reduce unsupported visual claims, providing interpretable and verifiable diagnostic reasoning.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The novelty of the proposed CARE framework is limited, as it mainly adopts prompt engineering and lacks conceptual and methodological innovation.
2.The task difficulty is overstated. Some disease samples can be visually distinguished by simple features such as color in Mel vs APN Fig1(a).
3.The baseline selection is inappropriate and unfair, mixing models with large performance gaps and different capabilities. Gemini3 > GLM> Qwen3
4.The multi-agent system is not a genuine agent architecture but only uses the same MLLM with different role‑specific prompts, and the agent concept is overstated.
5.The experimental setting is oversimplified and clinically unrealistic, with strict binary classification and no clinical context.
6.The overall performance is still insufficient for clinical use, and the generalization ability is not fully verified.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the paper addresses a clinically meaningful task and presents the CARE framework with observable performance improvements, it suffers from limited novelty, overstated task difficulty, inappropriate baseline comparisons, overclaimed multi-agent design, unrealistic experimental settings, and insufficient clinical performance. These critical shortcomings lead to my recommendation of Weak Reject.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper studies a meaningful and relatively underexplored setting: zero-shot differentiation of visually confounded disease pairs from images alone. It proposes CARE, a training-free three-agent framework in which two disease-specific agents generate opposing evidence and a judge performs image-grounded adjudication. The experiments show consistent gains over the Gemini-3-Flash baseline and over matched self-check / majority-vote variants, although the overall performance remains well below clinical-use level.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper’s focus on visually hard-to-separate disease pairs with different management implications is well motivated. 2.For methods, the role split between two specialist agents and an image-grounded judge is intuitive, and the qualitative examples help illustrate what the adjudication step is doing. 3.The empirical section is reasonably solid for a pilot study. The paper evaluates two modalities, compares against several single-agent baselines, and includes compute-matched ablations.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The evaluation setting is controlled, which is reasonable for a pilot study, but it is still somewhat artificial. The XOR formulation simplifies the task and does not reflect real co-occurrence, especially for edema and pneumonia. The paper acknowledges this limitation, but it still weakens the clinical interpretation of the results. 2.The idea is promising, but the current “among the first” wording feels a bit too broad. I think the paper would benefit from a clearer discussion of nearby work on medical agentic reasoning and robustness, especially recent work such as ‘PASS’ (AAAI-26 oral) [1], which studies interpretable and adaptive chest X-ray reasoning through probabilistic agentic workflows, and the MICCAI’25 paper ‘Knowing or Guessing’ [2]. These works do not invalidate the paper’s exact setting, but they do make the novelty boundary look narrower than currently stated. A more precise claim around training-free, zero-shot, image-only contrastive adjudication for visually confounded differential diagnosis would read as stronger and more credible. 3.Reproducibility is fair but not yet strong. The overall framework is understandable, but for a prompting-based method I would still like clearer reporting of prompts, inference settings, and some of the curation details. 4.The chest X-ray result is encouraging, but not fully convincing yet. CARE improves over Gemini-3-Flash, but still remains below Gemini-3-Pro on that task, so the practical benefit of the additional multi-agent overhead is not yet fully clear.
[1] Feng et al., PASS: Probabilistic Agentic Supernet Sampling for Interpretable and Adaptive Chest X-Ray Reasoning, AAAI 2026 Oral. [2] Jiang et al., Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning, MICCAI 2025.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
NA
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
Overall, I found the paper thoughtful and reasonably careful in its claims. I suggest authors to tighten the related-work positioning and narrow the novelty claim a bit. I think the paper would read better if it explicitly acknowledged adjacent work on agentic medical reasoning and robustness, then emphasized more clearly what is specific here: the training-free, image-only, zero-shot contrastive adjudication setup for visually confounded disease pairs.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a weak accept for me. Although the novelty is not yet positioned sharply enough relative to nearby work, and the current evidence is still limited to controlled proxy tasks. However, the paper addresses a good problem, the method is neat and promising, and the empirical gains are real.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
After the rebuttal, I am satisfied that the authors have addressed my main concerns, especially regarding the novelty boundary, data curation, and the practical value of CARE. Some limitations remain, particularly the simplified binary setting and the need for clearer reproducibility details, but I view them as acceptable for a pilot study and support acceptance.
Review #3
- Please describe the contribution of the paper
The paper introduces CARE, a training-free multi-agent framework designed to address the clinical challenge of distinguishing between visually similar diseases, such as melanoma and atypical nevi, in a zero-shot setting. By structuring a “contrastive adjudication” process where two specialized agents argue for opposing diagnoses while a third judge agent verifies their claims against the original image, the study demonstrates an 11-percentage-point accuracy gain on dermoscopy data compared to single-agent baselines. Beyond the methodological contribution, the authors provide a rigorous benchmark across two imaging modalities, revealing that while such structured reasoning significantly reduces unsupported “hallucinated” claims, the overall performance of current multimodal LLMs remains below the threshold required for safe clinical deployment.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The CARE system introduces a structured disagreement mechanism that effectively mimics the diagnostic process of human experts. Unlike standard self-reflection or majority-voting methods, CARE forces agents into mutually exclusive roles (one arguing for Disease A, another for Disease B), which successfully surfaces inconsistencies and reduces overconfident “hallucinated” claims. The study specifically focuses on two clinically significant proxy tasks: melanoma vs. atypical nevus and pulmonary edema vs. pneumonia. These pairs represent a “hard-to-separate” frontier where visual features are highly overlapped, making them an ideal and rigorous testbed for evaluating the reasoning limits of multimodal LLMs. The method is entirely prompt-based and requires no task-specific fine-tuning or additional annotated data. This makes the framework highly scalable and applicable to other medical imaging domains where high-quality labeled data for training might be scarce.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The authors note the exclusion of congenital and recurrent nevi because these subtypes are “often very difficult without clinical context”. However, as the paper’s title explicitly asks if agents can distinguish “visually hard-to-separate” diseases, manually removing the most challenging cases effectively defines a “simplified difficulty.” The authors should clarify if this exclusion intentionally shielded the zero-shot agents from the most difficult edge cases and whether the 11-percentage-point gain would significantly diminish if these cases were included.
To mitigate class imbalance, atypical nevi were subsampled to match the melanoma count (from 399 original cases to 257). The authors must clarify the specific methodology used for this subsampling (e.g., was it purely random?). Given the “visually confounded” nature of the task, it is critical to discuss whether this selection process introduced any bias or altered the prevalence of hard-to-distinguish cases compared to the original distribution.
The study employs a forced exclusive-or (XOR) setting for diagnosis. In real-world clinical practice, conditions like pulmonary edema and pneumonia frequently co-occur. By forcing a binary choice, the evaluation may artificially inflate accuracy and mask potential conflicts or “hallucinations” the model might encounter when dealing with comorbidities. The authors should discuss whether the Judge Agent should be granted the option of “both” or “neither” to better reflect clinical truth.
While “Blind-CARE” and “Self-Check” variants were tested , the current results do not fully isolate whether the performance gain stems from “contrastive reasoning” or simply the inherent capabilities of a superior judge. A more direct baseline (e.g. a single Gemini-3-Flash model performing a direct diagnosis without any debate assistance) should be emphasized for comparison. If the marginal gain of CARE over a standard single-call model is small, the added computational cost of a three-agent architecture may not be justified.
The qualitative analysis focuses on successful error corrections by the Judge Agent. To provide a balanced view of the system’s limitations, the authors should present cases where “reverse interference” occurred: situations where the two disease-specific agents provided such “persuasive” yet incorrect arguments that they successfully misled the judge.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend a 4 for this submission. The paper presents a timely and well-structured pilot study on using multi-agent MLLMs to distinguish visually confounded medical images, such as melanoma versus atypical nevi. The primary strengths lie in the CARE framework’s novel use of contrastive adjudication, which mimics expert reasoning to mitigate single-agent overconfidence and reduces “hallucinated” evidence, yielding a significant 11-percentage-point accuracy gain on dermoscopy data. However, the score is tempered by methodological concerns regarding the exclusion of the most challenging disease subtypes (congenital and recurrent nevi) and the use of a forced-choice XOR setting. These choices may simplify the problem space and artificially inflate the perceived effectiveness of zero-shot reasoning compared to real-world clinical complexity, where comorbidities frequently co-occur.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have adequately addressed my comments. I recommend acceptance.
Author Feedback
We thank all reviewers for constructive comments and for recognizing the forward-looking nature of this study. We address main concerns below:
1.Limited novelty (R1). CARE is not merely a prompt design, but a methodology of explicitly constructing disagreement. By doing this in a binary diagnosis setting, the system can comprehensively consider both sides before making a final decision. Importantly, it doesn’t require post-training or fine-tuning, with acceptable inference overhead.
2.Below-clinical performance (R1,R2). This study is not meant to claim clinical readiness, but to expose this important yet unsolved problem to the community. We have highlighted this point in Sec 3.2 and Sec 4.
3.Overstated task difficulty and agent concept (R1). We disagree with R1’s claim that melanoma (MEL) and atypical nevus (ANV) can be distinguished “simply by color.” In dermoscopic imaging, color is well known to be an unreliable and highly variable feature due to camera white balance, illumination temperature, etc. MEL vs. ANV remains challenging [15], and reducing it to color greatly oversimplifies the task. Moreover, we have clearly defined “agent” as “systems that can receive observations and perform goal-directed actions” based on [2]. Agent does not necessitate tool-use and RAG, it is the observation-action that really matters.
4.Baseline choice (R1, R3). We believe R1 has misinterpreted Table 1.The models listed there are not direct baselines for CARE, but rather as a benchmark survey to evaluate how current MLLM-based agents perform in this setting. For both R1 and R3, the actual comparison validating CARE is between Gemini-3-Flash (Baseline) and Gemini-3-Flash (CARE), where the underlying MLLM remains identical and only the inference framework changes. Therefore, the reported gains are not derived from comparing against weaker models, but from controlled comparisons under the same backbone model.
5.Oversimplified setting (R1/R2/R3). (1) absence of clinical context. Currently no publicly dataset provides paired imaging and clinical context. Therefore, as stated in the Discussion, our experiments should be viewed as imaging-only proxy tasks. Note that MLLM-based agents can naturally incorporate additional clinical context as textual input, allowing seamless extension to richer multimodal settings. (2) Simplified binary edema vs. pneumonia formulation. This pilot study is to establish that distinguishing visually confounded diseases remains unsolved for MLLM-based agents. As agents already struggle under this binary setting, more realistic non-exclusive formulations would likely be even more challenging. It doesn’t undermine our conclusion.
6.Practical benefit (R2). CARE is designed as a plug-and-play framework rather than a model-specific architecture. Applying CARE to Qwen3-VL-32B improved MEL vs. ANV F1 by 3.8 percentage points and increased the Youden index from 0.347 to 0.426.In principle, CARE could also be applied to Gemini-3-Pro. However, Gemini-3-Pro is limited to 250 API calls per day. An entire CARE inference would require 8 days for dermoscopy and 26 days for chest X-ray evaluation. Due to these constraints, we treated Gemini-3-Pro as a single-agent upper-bound reference rather than another CARE backbone.
7.Related work (R2). We thank R2 for pointing out these works and will cite them. Our work differs from these studies in focus. PASS investigates agents with tool-use mechanisms, while “Knowing or Guessing” proposes training recipes. In contrast, CARE focuses on training-free inference-time contrastive adjudication.
8.Data curation (R3). For ANV, we performed controlled subgroup-aware random sampling. We also apologize for the ambiguity in the manuscript regarding congenital and recurrent nevi. These subtypes were excluded because they are often considered theoretically indistinguishable from melanoma based on dermoscopy alone. The exclusion was to improve rigor rather than simplify the benchmark.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
After reviewing this manuscript together with the reviewers’ evaluations, I note that the paper received mixed but generally constructive feedback. The reviewers recognize the clinical relevance of distinguishing visually confounded diseases in a zero-shot setting, as well as the potential value of the CARE framework for contrastive adjudication and reducing unsupported claims. At the same time, several important concerns remain, including limited methodological novelty, possible overstatement of the multi-agent formulation, simplified binary/XOR experimental settings, baseline fairness, and insufficient evidence of clinical realism and generalization. Based on my assessment, these issues would benefit from clarification and targeted response by the authors. I therefore recommend that the paper proceed to the rebuttal phase for further consideration.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper is recommended for acceptance based on its solid contribution as a timely pilot study and the thoroughness of the post-rebuttal clarifications. The authors successfully addressed initial concerns regarding baseline fairness and task framing.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
the reviewers acknowledges the method is clinical meaningful, timely, with well established empirical gains. the authors addresses the points raised by reviewers well; baseline comparison, the novelty boundary, data curation, and the intended scope as an imaging-only pilot.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This paper proposes CARE, a training-free three-agent framework for zero-shot differentiation of visually confounded disease pairs. The setting is meaningful and underexplored, and the core idea of constructing explicit disagreement for contrastive adjudication is well motivated.
After rebuttal, both Reviewer #2 and Reviewer #3 are satisfied and explicitly recommend acceptance. The main criticisms raised by Reviewer #1 concern limited novelty, baseline fairness, oversimplified experimental settings, and below-clinical performance. The authors have provided substantive responses: they clarified that CARE is a methodology of constructing disagreement rather than mere prompt engineering, which is a valid conceptual contribution; they explained that the baseline comparison is conducted under identical backbone models, thus the gains are fairly attributed; they acknowledged the simplified binary setting and the imaging-only nature as limitations of this pilot study, while noting that the framework can naturally incorporate clinical context in future work; and they were transparent that the current performance is not clinical-ready but aims to expose an unsolved problem to the community. The clarification on the definition of agent is also reasonable.
The remaining concerns, including the binary task formulation and the absence of clinical context, are acknowledged as limitations and are acceptable for a pilot study that opens a new direction. I recommend accept. For the camera-ready version, the authors should: (1) clearly state the limitations regarding binary classification and imaging-only evaluation in the abstract and conclusion; (2) provide the exact definition of agent adopted in this work early in the manuscript; (3) ensure full reproducibility details, including prompts and inference configurations, are documented; and (4) incorporate the newly cited related work as promised in the rebuttal.