Abstract
A benchmark score cannot say whether a medical foundation model reads a disease or merely a correlate of it. The common recourse is attention faithfulness, which scores a model’s attention against expert annotations and reads agreement as evidence of grounding. On coronary artery calcification (CAC), that agreement conflates patient-specific evidence with the patient-independent anatomical prior. We freeze three ViT-L encoders and train a shared attention-MIL head on the features of each. Their detection intervals overlap, whereas faithfulness ranges from 0.554 to 0.909; read against the chance baseline of 0.5, that range would suggest that only the 3D medical encoder localises the disease. Calcium occurs throughout the thorax. The Agatston score, however, counts only what lies in the coronary arteries, so every annotated lesion sits on the coronary tree, and every volume shares a resampling grid. A patient-independent prior built in the encoder’s own instance space therefore reaches 0.937 against the 3D encoder’s 0.909, and no encoder beats it. Replacing a patient’s attention with the leave-one-out attention prior changes the 3D encoder’s faithfulness by +0.001. In this cohort, the attention–annotation agreement conventionally reported as faithfulness is largely explained by where lesions typically occur rather than by additional localisation of an individual patient’s lesion. This does not mean the 3D encoder ignores calcium. Grafting a donor lesion into a CAC-negative volume raises its predicted probability by 0.733 and flips 85% of those patients, with a response already visible at 4 voxels, while the 2D encoders show weaker responses. Where lesion location is anatomically constrained, as for CAC, faithfulness should be reported alongside its gain over a patient-independent prior, rather than interpreted against chance alone, so that what attention adds beyond anatomical expectation is visible, and grounding claims should be complemented by controlled intervention-based evidence.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MI4MedFM_026.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=zTSKKWKach
BibTex
@InProceedings{BatAsh_The_MICCAISAT2026,
author = { Bathineni, Ashajyothi AND Mishra, Siddhant AND Diederichsen, Marie Zöga AND Diederichsen, Axel AND Gogineni, Vinay Chakravarthi AND Blanes-Vidal, Victoria},
title = { { The Metric, Not the Model: Anatomy Confounds Attention Faithfulness in Coronary Calcium } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17261},
month = {pending},
page = {pending}
}