Abstract
Vision-language models, such as contrastive language-image
pre-training (CLIP)-based approaches, have reached state-of-the-art (SOTA)
results in medical artificial intelligence. However, recent work reveals that
CLIP-based models remain vulnerable to shortcuts. We investigate how
real-world shortcuts manifest across different layers of the medical CLIP-
based model, MedCLIP, and its vision encoder, a frozen ResNet-50. We
attach 17 linear classification probes to the intermediate layers of the
ResNet-50 and train them on three different dataset configurations and
targets: NIH-CXR14 (pneumothorax) and PadChest (cardiomegaly and
pneumothorax). This setup allows us to observe model behaviour during
evaluation using subgroup-based calibration and layer-wise confidence
curves. We find that the final linear probes achieve a high AUROC but
poor calibration in the models. The layer-wise confidence analyses sug-
gest that shortcuts emerge at different depths. Patterns consistent with
localised shortcuts, such as drains, appear at later layers, while patterns
consistent with diffuse shortcuts, such as scanner-specific noise patterns,
emerge earlier, aligning with previous work. Finally, we conduct a manual
analysis of the images, which reveals data quality issues in both NIH-
CXR14 and PadChest. Our findings underscore that even SOTA models
remain vulnerable to shortcuts, and the need for high-quality and well-
annotated datasets to draw solid conclusions. Code can be found on our
GitHub: https://github.com/nikodice4/MedCLIP_shortcuts.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/FAIMI-BRIDGE-EPIMI_008.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=zxnWbkVbZe&nesting=2&sort=date-desc
BibTex
@InProceedings{PedNik_Look_MICCAISAT2026,
author = { Pedersen, Nikolette AND Sydendal, Regitze AND Cheplygina, Veronika AND Sourget, Théo},
title = { { Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17279},
month = {pending},
page = {pending}
}