Abstract
Zero-shot medical vision-language models (VLMs) are proposed as plug-and-play triage tools for chest radiography, yet their probabilities are miscalibrated. Prior evaluations established this only qualitatively or in binary, single-label settings; we are not aware of one that sweeps calibration-set prevalence. We present a standardized multi-label benchmark for five chest X-ray CLIP backbones and five co-occurring CheXpert findings. It sweeps per-finding calibration prevalence from 1% to 20%, compares post-hoc and trainable calibrators with a transductive adaptation stress test, and reports macro ECE, smooth ECE, and AUROC. Prevalence reaches calibration error almost entirely through the calibrator’s intercept: a map without one is insensitive to the calibration prior yet miscalibrates rarer findings, while a bias term removes and reverses that trend. Under the deployment regime of a fixed, separate calibration set, the bias term reaches only a floor near 0.08-0.10 that no post-hoc map we test, parametric or nonparametric, breaches. In a per-finding Platt analysis, prevalence matching alone closes about half of the gap to an in-distribution refit on the three backbones that discriminate. Such a refit removes most of the floor, but deployment cannot. That floor is independent of discrimination, which spans macro AUROC 0.54-0.89, so a near-random backbone reaches the same floor as the best by predicting near the base rate. Low calibration error therefore does not certify a usable model: our stress test posts ECE 0.029 while ranking below chance, and a second failure mode mis-ranks at an ECE indistinguishable from a well-ranking model. Report calibration error with a discrimination metric.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/UNSURE2026_028.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/UNSURE2026_028_supp.pdf
Link to Open Review
Open Review Page: https://openreview.net/profile?id=~Tianyi_Zhang22
BibTex
@InProceedings{ZhaTia_WellCalibrated_MICCAISAT2026,
author = { Zhang, Tianyi AND Liu, Zixuan},
title = { { Well-Calibrated but Unusable: Calibration Error Does Not Certify Zero-Shot Chest X-Ray VLMs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17260},
month = {pending},
page = {pending}
}