Abstract
Modeling inter-rater ambiguity, the disagreement among radiologists annotating the same image, is central to trustworthy medical image segmentation. The rise of agentic AI suggests a tempting recipe: prompt a vision language model (VLM) with distinct radiologist personas, let each propose a mask, and treat the resulting spread as a model of clinical disagreement. We assessed whether such training free persona committees actually reproduce human inter rater ambiguity on lung nodule segmentation using the LIDC dataset. Each persona proposes a bounding box that a promptable segmenter (MedSAM) refines into a mask and the per pixel entropy of the committee forms an ambiguity map. We evaluate three backends, a small general VLM (Qwen), a frontier VLM (Claude), and a medical VLM trained on CT (MedGemma), and we stratify disagreement into boundary cases, where all raters mark the lesion, and existence cases, where some raters see nothing. A parameter free geometric null based on distance to the consensus boundary predicts human disagreement better than every committee, reaching AUROC 0.87 versus 0.53 on boundary cases and 0.64 versus 0.63 on existence cases. Committee diversity is also miscalibrated as Claude personas agree much more than radiologists on existence cases, with Dice 0.70 versus 0.39, while MedGemma personas disagree much more on boundary cases, with Dice 0.47 versus 0.79. These results show that off the shelf persona committees do not capture clinical ambiguity and that effective methods require calibrated diversity and explicit abstention.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedAgent_043.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=dYHZZtUPWW
BibTex
@InProceedings{AlMd_Too_MICCAISAT2026,
author = { Al Banna, Md Hasan AND Rahman, Md Maklachur AND Hammond, Tracy},
title = { { Too Tight or Too Loose: How VLM Persona Committees Miscalibrate Inter-Rater Ambiguity in Medical Image Segmentation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17263},
month = {pending},
page = {pending}
}