Abstract
“Medical vision–language models (VLMs) are increasingly proposed for
discriminative, safety-critical tasks that require reporting not only what is
present in an image but when an expected structure is absent. We test whether
organ-presence answers are visually grounded or driven by an anatomical prior,
using a manipulation-free protocol: unaltered abdominal computed tomography (CT)
slices from BTCV, with ground truth read directly from voxel labels. We adapt POPE-style polling to
volumetric anatomy, querying absent organs under random, popular, and adversarial
negative sampling. Across six VLMs spanning medical specialization, scale, and
architecture, sensitivity stays near ceiling while specificity collapses as the
prior strengthens; under adversarial sampling MedGemma-27B rejects the most
anatomically expected absent organ only 4.7% of the time. A base-vs-medical comparison (Gemma3 vs. MedGemma at 4B and 27B,
matched in
architecture and scale) shows the bias is already present in the base model and is
mitigated, not caused, by medical adaptation. Reasoning-first structured output
appears to recover specificity, but the gain is an artifact of abstention:
Qwen2.5-VL reaches 0.642 binary specificity while its responses carry uncertainty
cues on 94% of queries,
and falls to 0.069 on confident answers. Binary presence scoring therefore
over-credits abstaining models; confident-answer specificity is required to
measure grounding.”
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/SAFER_014.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=IYZEAeCpdy
BibTex
@InProceedings{NazAle_Priors_MICCAISAT2026,
author = { Nazarov, Alexander AND Mayer, Arnaldo},
title = { { Priors Over Pixels: Present-Bias in Organ-Presence Grounding for Medical VLMs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17261},
month = {pending},
page = {pending}
}