List of Papers Browse by Subject Areas Author List
Abstract
Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural imaging (e.g., MRI), selectively add sequences such as FLAIR or T2 to refine the differential, and reserve molecular imaging (e.g., amyloid-PET) for cases that remain uncertain after standard evaluation. Consequently, patients are observed with heterogeneous and often incomplete modality subsets. However, most current AI models assume fixed data modalities as the model inputs. In this paper, we present BrainAnytime, a unified pretraining framework pretrained on 34,899 3D brain scans from five datasets that support brain image analysis under arbitrary modality availability spanning multi-sequence MRI and amyloid-PET. A single model accepts whatever imaging is available, from a lone T1 scan to a full multimodal workup. Pretraining learns structural-molecular correspondences between MRI and PET via cross-modal distillation (RCMD) and prioritizes disease-vulnerable anatomy via atlas-guided curriculum masking (PACM), all within a shared 3D masked autoencoder (Multi-MAE3D). Across four downstream tasks and five clinically motivated modality settings, BrainAnytime largely outperforms modality-specific models, missing-modality baselines, and large-scale brain MRI pretrained foundation models on most modality settings. Notably, it surpasses the strongest missing-modality baselines with relative improvements of 6.2% and 7.0% in average accuracy on CN vs. AD and CN vs. MCI classification, respectively. Code is available at https://github.com/SDH-Lab/BrainAnytime.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3498_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/SDH-Lab/BrainAnytime
Link to the Dataset(s)
N/A
BibTex
@InProceedings{YanGua_BrainAnytime_MICCAI2026,
author = { Yang, Guangqian AND Ding, Tong AND Hou, Wenlong AND Xun, Yue AND Du, Ye AND Niu, Qian AND Wang, Shujun AND the Alzheimer’s Disease Neuroimaging Initiative},
title = { { BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16880},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper introduces BrainAnytime, a unified pretraining framework designed to handle brain image analysis under arbitrary modality availability (T1, T2, FLAIR, and Amyloid-PET). The main contributions include: 1) Multi-MAE3D: A 3D multimodal masked autoencoder with a shared Transformer encoder that processes any subset of modalities without architectural changes. 2) Reciprocal Cross-Modal Distillation (RCMD): A distillation objective that explicitly learns structural-molecular correspondences between MRI and PET. 3) Pathology-Aware Curriculum Masking (PACM): An atlas-guided masking strategy that prioritizes disease-salient anatomy during the reconstruction process.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Unlike models that assume fixed inputs, this work is motivated by the clinical modality escalation pathway, making it highly relevant for real-world diagnostic workflows where data is often incomplete. The model is pretrained on a massive dataset 3D scans, ensuring high robustness and generalization. It provides a novel way to incorporate prior medical knowledge into a self-supervised learning framework.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- While the framework is titled “BrainAnytime,” the PACM importance weights are specifically tuned for Alzheimer’s Disease using Braak staging. Its generalizability to other neurodegenerative diseases or brain tumors, where different regions are salient, is not explored.
- The PACM strategy requires registering input volumes to the MNI152 space to apply the atlas. The authors do not discuss how registration errors—which are common in patients with severe cortical atrophy—might propagate and affect the importance scoring.
- In RCMD, T1, T2, and FLAIR are pooled into a single group. Since these sequences capture distinct pathological features, grouping them might dilute sequence-specific information that could be useful for cross-modal alignment.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses a significant and practical problem in medical AI—missing data—with a technically sound and clinically inspired methodology. The integration of anatomical priors into a masked autoencoder is both novel and effective. Given the large-scale evaluation and clear performance gains over state-of-the-art baselines, this work provides a valuable contribution.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a multimodal brain imaging pretraining framework that supports flexible combinations of input modalities, including multi-sequence MRI and PET. The method is designed to: (1) accommodate arbitrary modality combinations as input, (2) propose cross-modal feature alignment between multi-sequence MRI and PET to capture structural–molecular correspondence, and (3) emphasize pathology-relevant regions/patches through a pathology-aware learning strategy.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is well organized and clearly presented The proposed pathology-aware curriculum masking module is novel and well-motivated. The authors conducted thorough comparison and ablation experiments.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
See additional comments below
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
For some downstream tasks, incorporating additional modalities does not consistently improve performance compared to using fewer modalities or even a single modality. As shown in Table 2, the classification and MMSE prediction performance decreases when adding T2 (e.g., T1 vs. T1+T2+FLAIR, and T1+FLAIR vs. T1+T2+FLAIR). It would be helpful if the authors could provide additional insights into this.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The idea is novel, the paper is well organized, and the experiments are thorough.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper presents a self-supervised pre-training framework on large-scale multimodal brain images that accommodates missing modalities, and demonstrates superior performance by leveraging incrementally more modalities for AD classification, cognitive score and age prediction tasks.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper has fairly reasonable clinical formulation and relevance of missing modalities, and the scale and diversity of data and extensive benchmarking also stand out. The PACM and RCMD modules are also clinically motivated (anatomy relevant prior for masking, bridging gaps between MRI and PET). There’s also consistently high performance over strong missing modality baselines and inclusion of ablation studies.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
First, the paper does not explain why tau-PET as a core AD biomarker is not included. Second, while pre-training is on diverse datasets, evaluation is only on ADNI, which has similar distribution to training datasets. Inclusion of more out-of-distribution independent external datasets could be helpful to strengthen the claim of the generalizability. Third, the paper does not explain or why using RCMD is optimal over many grounded alternatives (CLIP style, token-level alignment, masked cross-reconstruction (standard in MultiMAE)), nor ablations on them, since there’s no ablation on the effectiveness on specifically cross-modal alignment when PET is not present. Fourth, it’s also not sure whether dynamic score is better than static score in PACM. Fifth, label efficiency is better demonstrated with comparison with baselines.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation, method is generally well-grounded, results are through and have good performance and interpretations.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
N/A
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper introduces BrainAnytime, a unified pretraining framework for brain image analysis with arbitrary modality availability using Multi-MAE3D, Reciprocal Cross-Modal Distillation, and Pathology-Aware Curriculum Masking. R1 highlights the significant and practical problem addressed and the valuable contribution. R2 praises the novel idea, well-organized paper, and thorough experiments. R3 finds the motivation and method well-grounded with good performance. The paper addresses the important problem of handling arbitrary modality availability in brain image analysis, and the proposed framework with multi-modal masked autoencoding, cross-modal distillation, and pathology-aware masking represents a substantial methodological contribution. The thorough evaluation across multiple modalities and downstream tasks are also good points for the paper.