Abstract
Multimodal large language models (MLLMs) can process both text and images, which raises the question of whether they can perform clinical ultrasound interpretation without any task-specific training. We evaluate two off-the-shelf MLLMs, GPT-5.2 and Gemini-3-Flash, on two ultrasound tasks with different visual reasoning demands: left ventricular ejection fraction (LVEF) estimation from echocardiography, and breast lesion classification. In a qualitative analysis, the models cannot reliably estimate LVEF, failing at temporal reasoning and assessing left ventricular size. Breast lesion classification is more promising, and we study it in two evaluation settings. In a single-request setting, the model answers in a single call, and we vary the prompting (direct vs. chain-of-thought) and the input (with or without a segmentation overlay). Separately, we test a multi-agent decomposition that composes these same primitives into a structured workflow with multi-reader consensus and a memory of past cases, mimicking a radiologist’s descriptor-based workflow. In the single-request setting, the two models react differently: GPT-5.2 improves with CoT prompting (up to 74.6% binary BI-RADS accuracy) but is largely unaffected by the overlay, whereas Gemini-3-Flash improves with the overlay but changes little with CoT. No configuration lifts malignancy prediction above the 50% chance level (52.0-57.5%), and the multi-agent pipeline does not beat single-request prompting end-to-end. When the diagnosis agent is given ground-truth lesion features, binary BI-RADS accuracy rises to 78.0%. When the model extracts the features itself, accuracy drops to 65.5%, since the feature extraction errors are concentrated on the most decisive features (35.5% margin and 26.5% shape mismatch rate). This locates the main bottleneck at perception rather than reasoning. Even with ground-truth lesion features, malignancy accuracy reaches only 59.0%, well below the fine-tuned ResNet-50 and Qwen2.5-VL-3B models (73.8% and 77.9%), suggesting a second limitation in the human-style descriptor representation itself. We conclude that zero-shot MLLMs are not yet reliable for ultrasound diagnosis, and that perception, not reasoning, is the main wall.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedAgent_016.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=AjMATRDMX8
BibTex
@InProceedings{DevChr_Disentangling_MICCAISAT2026,
author = { Devin, Christian AND Alikhani, Alireza AND Azampour, Mohammad Farid AND Navab, Nassir},
title = { { Disentangling Perception and Reasoning in Zero-Shot Multimodal LLMs for Ultrasound Diagnosis } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17263},
month = {pending},
page = {pending}
}