Abstract
Public chest X-ray repositories are widely used to train medical AI systems, yet their labels are typically extracted from radiology reports rather than verified directly on images. As a result, repository labels are often treated as image-level ground truth without validating whether they reflect what is actually visible in the radiograph. We introduce Repository Supervision Auditing (RSA), a framework that evaluates repository-derived labels against expert image-level annotations before model development. Using cardiomegaly in MIMIC-CXR as a case study, RSA compares repository labels with radiologist-reviewed image annotations, characterizes disagreement sources, and builds a curated cohort for deployment-oriented evaluation. Repository-derived cardiomegaly labels showed near-zero agreement with expert image-level assessment, identifying only 1% of expert-confirmed cases. Most discrepancies resulted from non-mention rather than explicit report negation, with expert-confirmed cardiomegaly identified in nearly half of studies assigned a repository-derived No Finding label. Using the resulting expert-curated cohort, a DenseNet121 model achieved a test ROC-AUC of 0.853. These findings show that repository labels may not reliably represent image-level truth and highlight supervision auditing as a critical step for developing trustworthy medical imaging AI.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/EMERGE_007.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=XgT04TBiKB
BibTex
@InProceedings{AguYes_When_MICCAISAT2026,
author = { Agudelo-Londoño, Yesika Alexandra AND Pino-Román, Jhon Wilmer AND Carrera Rodríguez, Brahian AND Castañeda-Bedoya, José Miguel AND Gómez-López, Juan Pablo AND Puche-Sarmiento, Aura C. AND D’Souza, Niharika S. AND Osorio-Valencia, Juan Sebastian AND Duque-Grajales, Jon E. AND Suárez-Revelo, Jazmín Ximena AND Vélez-Arango, Jorge Mario AND Castrillón, Gabriel},
title = { { When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17266},
month = {pending},
page = {pending}
}
