Abstract
Open CXR vision-language models (VLMs) increasingly display decoder cross-attention as an overlay of “where the model looked”; whether it corresponds to radiologist-localisable regions is unexplored. We audit mean cross-attention against radiologist boxes for four released checkpoints (MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, LLaVA-1.5) on three datasets (VinDr-CXR, RSNA Pneumonia, MS-CXR). (1) MAIRA-2 > MedGemma > LLaVA pair holds on all three datasets and a held-out Pneumothorax cohort, but against a class-conditional anatomical atlas every model’s typical-case attention is below the prior (own-prior gaps -0.06 to -0.27 on VinDr); a scale-invariant enrichment ratio (5.5/2.8/0.9/1.2x) preserves the order, and on RSNA MAIRA-2 is only +0.008 above a dataset-mean-bbox oracle. (2) Only where the prior is far from its class centroid do MAIRA-2 (+0.14) and MedGemma (+0.13) surpass it and the LLaVA pair do not, so genuine image-driven localisation lives in the two medical models alone; attention concentration predicts their per-case faithfulness (AUROC 0.65-0.68). (3) MedGemma reproduces 0.49 of its pattern on black images and its free-generation overlay collapses to 0.556/0.459/0.571; MAIRA-2’s is image-scale invariant (real-vs-flip cosine 0.972), paraphrase-invariant (|dAUC|<0.013). (4) Substituting MAIRA-2’s L2 map into MedGemma’s L17 betters AUC +0.153, but a held-out atlas reproduces this (0.896 vs 0.901), so the active ingredient is anatomical concentration. (5) A two-radiologist reader study (124 cases) calibrates the proxy: plausibility ratings track Alignment-AUC (Spearman rho=0.54) and the random control is rated worst, yet radiologists rate the plausible, prompt-anchored MedGemma and the anatomical prior above the more box-faithful MAIRA-2, confirming that overlay plausibility is not faithfulness. We release per-case data and code, and recommend reporting grounding faithfulness as a prior-residualised, finding-stratified off-prior gap plus enrichment ratio, validated against radiologist judgement, for new CXR VLMs.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/iMIMIC_010.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/iMIMIC_010_supp.pdf
Link to Open Review
Open Review Page: Not Available
BibTex
@InProceedings{FaiSye_How_MICCAISAT2026,
author = { Faizan, Syed AND Vasamsetti, Mahesh AND Shivam Dilipbhai, Kotak AND Madhavi Latha, Gundamaraju AND Krishnamurthy, Parameswari},
title = { { How Well Do Chest X-Ray VLM Attention Overlays Match Radiologist Boxes? A Cross-Model Audit and Radiologist Reader Study } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17279},
month = {pending},
page = {pending}
}