Abstract

Universal representation learning for brain MRI is hindered by the high heterogeneity of neurological pathologies. Despite the promise of vision–language (V-L) pretraining, current methods rely primarily on implicit textual supervision and overlook strong diagnostic priors in radiology reports. To address this, we propose a Joint Vision-Language-Diagnosis Alignment framework that integrates cross-modal contrastive learning with multi-label diagnostic supervision. Unlike conventional V-L pretraining, this framework explicitly enforces semantic granularity at different levels: aligning global report semantics while sharpening class separability through diagnostic priors. We demonstrate this on a large-scale dataset of 50,053 routine clinical MRI–report pairs. Extensive experiments show that incorporating diagnostic supervision substantially improves semantic separability and cross-modal alignment: under linear probing, Macro F1 nearly doubles on the 47-class brain abnormality classification task (from 0.2723 to 0.5275), while Image-to-Text R@1 improves by over 17% in cross-modal retrieval. The learned representations also exhibit improved downstream transfer across distinct pathological settings, yielding AUC gains of up to 6.64 and 5.61 points over pure contrastive learning on focal lesion recognition (BraTS 2018) and diffuse neurodegeneration classification (ADNI), respectively. These results indicate that incorporating explicit diagnostic priors effectively constrains the V-L solution space and promotes representations that support downstream brain MRI analysis.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/BrainWorks_008.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to Open Review

Open Review Page: https://openreview.net/profile?id=~Feng_Shi2

BibTex

@InProceedings{ZheQia_BrainMRICLIP_MICCAISAT2026,
        author = { Zheng, Qiankun AND Wu, Xintong AND Wu, Yaping AND Liu, Yuxiao AND Qiu, Zhaoyu AND He, Yichu AND Shi, Feng AND Wang, Meiyun AND Shen, Dinggang},
        title = { { BrainMRI-CLIP: Enhancing Brain MRI Representation Learning via Joint Vision-Language-Diagnosis Alignment } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 17254},
        month = {pending},
        page = {pending}
}


back to top