List of Papers Browse by Subject Areas Author List
Abstract
Current fMRI foundation models primarily rely on a limited range of brain states and mismatched pretraining tasks, restricting their ability to learn generalized representations across diverse brain states. We present Brain-DiT, a universal multi-state fMRI foundation model pretrained on 349,898 sessions from 24 datasets spanning resting, task, naturalistic, disease, and sleep states. Unlike prior fMRI foundation models that rely on masked reconstruction in the raw-signal space or a latent space, Brain-DiT adopts metadata-conditioned diffusion pretraining with a Diffusion Transformer (DiT), enabling the model to learn multi-scale representations that capture both fine-grained functional structure and global semantics. Across extensive evaluations and ablations on 7 downstream tasks, we find consistent evidence that diffusion-based generative pretraining is a stronger proxy than reconstruction or alignment, with metadata-conditioned pretraining further improving downstream performance by disentangling intrinsic neural dynamics from population-level variability. We also observe that downstream tasks exhibit distinct preferences for representational scale: ADNI classification benefits more from global semantic representations, whereas age/sex prediction comparatively relies more on fine-grained local structure.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1362_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/REDMAO4869/Brain-DiT
Link to the Dataset(s)
Multiple public neuroimaging datasets were used in this study, including HCP, ABCD, ABIDE, ADNI, NKI, ADHD-200, PPMI, NSD, StudyForrest, and others. Dataset access information and links are provided in the manuscript and project repository: https://github.com/REDMAO4869/Brain-DiT
BibTex
@InProceedings{XiaJun_BrainDiT_MICCAI2026,
author = { Xia, Junfeng AND Ye, Wenhao AND Pan, Xuanye AND Shen, Xinke AND Wang, Mo AND Liu, Quanying},
title = { { Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16894},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper pitches Brain-DiT, a massive multi-state fMRI foundation model. The authors trained a Diffusion Transformer on nearly 350k sessions across 24 datasets, covering resting, task, naturalistic, and disease states. It uses metadata-conditioned pretraining and aggregates features across different layers and timesteps for downstream tasks. They test it heavily on ID and OOD datasets, frozen-backbone transfers, and state-diversity ablations.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This is a paper with a huge workload. I like the scope: moving past single-state pretraining to build a truly universal multi-state representation is exactly what the field needs. Using a diffusion objective is a smart choice here, as it naturally gives you a nice hierarchy of features across timesteps. The experimental package is super robust. The fact that the frozen-backbone results are this strong really proves that the pretraining actually learned useful, generalizable representations, rather than just relying on fine-tuning downstream.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The elephant in the room is the metadata conditioning. You’re feeding the model age, sex, and disease info during pretraining, but then evaluating it on downstream tasks that predict those exact variables. It’s hard to untangle whether the downstream performance boost is coming from a universal representation or just side-information leakage from the pretraining phase. The unconditional ablation helps a bit, but it doesn’t entirely put the issue to bed. Also, since some of your ID benchmarks are drawn from the pretraining datasets, you need to be explicitly clear about how you guaranteed strict subject/site-level data separation.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I gave this a weak accept because it’s tackling a huge problem, the engineering is impressive, and the frozen-backbone results are super convincing. My only real hesitation is the fairness of the evaluation protocol due to the metadata conditioning.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper presents Brain-DiT, the universal multi-stage foundation model for fMRI data, that incorporates metadata-conditioned Diffusion Transformer. Paper proposes using diffusion process by incorporating gaussian noise and optional conditional embedding to incorporate multi-state fMRI to obtain generalized denoised fMRI later used for downstream tasks adaptation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Training uses multiple brain states, unifying almost 350 thousand sessions and 24 datasets containing different states, 2.Its the first model that includes metadata-conditioned pertaining 3.The model is outperforming other solutions in ID and OOD tasks. 4.The code and parameters are available, what increases reproducibility and future usage.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.the amount of models used for comparison is very limited 2.there are only 4 datasets used as evaluation and there is no description of how different they were from training ones
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.All dataset showed in the paper are highlighting that model outperformed others, how about remaining dataset, is it also a case, or are there any where the model is comparable wit others/ worse? if yes, why do you think it happened?
2.How datasets for OOD evaluation were selected?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Generally interesting paper, however it would be preferable to highlight how selected evaluation datasets were different/similar to the training ones and possibly perform ablation and train at least one more model with different combination.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper presents a foundation model for fMRI that is pretrained on a large and diverse collection of 24 datasets spanning multiple brain states, with the goal of learning transferable representations for downstream neuroimaging tasks. A key methodological contribution is a novel diffusion-based approach combined with metadata conditioning, which together enables state-of-the-art performance across the evaluated benchmarks.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper proposes a novel diffusion-based foundation model for fMRI with metadata conditioning, which is an interesting and original way to model heterogeneous brain imaging data across different brain states. Another major strength is the strong empirical performance: pretraining on 24 datasets is substantial in scale, and the reported state-of-the-art results suggest that the approach is effective in practice.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.More details on the downstream tasks and experimental setup are needed. In particular, it is unclear why different baselines were used across experiments and why not all datasets were included in the frozen-backbone evaluation, and in several cases it is also difficult to understand what the actual task is for each dataset.
2.Comparisons to scratch training are missing, and depending on the answer to the point above, comparisons to full fine-tuning or linear probing would also be important. Without these baselines, it is difficult to judge the benefit of the proposed strategy.
3.The evaluation would benefit from greater clarity. Accuracy is hard to interpret without reporting class distributions, and the table should cite the baseline methods and clarify whether they were trained by the authors or taken from public checkpoints.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I will increase my score after the rebuttal if the missing information is added, especially clearer details on the downstream tasks and experimental protocols, as well as more complete reporting of classification metrics.
At the same time, I note that some of these limitations may reflect the constraints of the MICCAI format (8 pages and no supplementary) rather than a lack of care by the authors. The paper covers a broad and ambitious experimental scope, and several aspects would simply benefit from more space for explanation.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors adressed most of my comments.
Author Feedback
We sincerely thank the reviewers and address the major concerns regarding metadata leakage, data separation, evaluation scope, and baseline clarity.
Metadata conditioning and leakage (R1). We clarify that there is no downstream metadata or label leakage. In every downstream task, the model uses only fMRI input; no metadata, including age, sex, diagnosis, site, or dataset labels, is used as downstream input. Metadata conditioning during pretraining is not a prediction shortcut, but an encoded condition vector used only for the denoising objective, helping the denoiser learn how fMRI signals vary across age, sex, and clinical groups. As described in Sec. 2.2 and Sec. 3.1, CFG-style condition dropout replaces metadata embeddings with an unconditional embedding (p_drop=10%), reducing reliance on metadata alone. Brain-DiT(uncond) removes metadata conditioning entirely during pretraining and is evaluated under the same fMRI-only downstream protocol. Its strong performance against representative baselines shows that diffusion-based denoising itself is a stronger pretraining proxy, while the improvement of Brain-DiT over Brain-DiT(uncond) reflects the additional benefit of metadata-guided denoising rather than downstream label leakage.
Data separation, evaluation protocol, and OOD scope (R1&R2&R3). We clarify that all ID evaluations use strict subject-level separation within each dataset/site. Subjects are split 7:1:2 before pretraining and downstream evaluation. For ID datasets overlapping with the pretraining corpus, held-out test subjects and their fMRI windows are strictly excluded from pretraining, downstream training, and model selection. For OOD evaluation, NKI, ADHD-200, and PPMI are held out at the dataset level from pretraining, tuning, and model selection to test cross-dataset shifts. In terms of evaluation scope, the manuscript reports 11 downstream evaluation settings, rather than only the 4 frozen-backbone settings. Table 1 reports full fine-tuning results over 7 settings, covering ID and OOD evaluations. Table 2 reports a 4-setting frozen-backbone linear-probing evaluation, also covering ID and OOD tasks, to assess representation transfer without updating the backbone. Thus, Table 2 is a transfer subset, not the full evaluation set. The actual tasks are: HCP binary sex classification; ABIDE, SALD, and NKI-AGE age regression; ADHD-200 binary disease classification; PPMI 3-class disease classification; and NKI-EDU 3-class education classification. Given 24 datasets and 7 comparison models/variants, we designed a representative suite rather than an exhaustive dataset-model grid, covering ID/OOD generalization, classification/regression, and full/frozen-backbone transfer.
Baseline selection and experimental protocol (R2&R3). Brain-DiT is an ROI-level model, so our full fine-tuning comparison focuses on ROI-level baselines, including BrainLM, BrainMass, and Brain-JEPA, to avoid confounding pretraining quality with voxel-level input granularity and end-to-end model capacity. In the frozen-backbone setting, the goal is to isolate transferable pretrained representations rather than adaptation capacity; therefore, we additionally include compatible voxel-level baselines. This shows that Brain-DiT learns strong transferable representations without dense voxel-level inputs and can outperform compatible voxel-level variants. For all baselines, we use the authors’ official implementations and public checkpoints, with downstream fine-tuning/linear probing under the same splits and protocol. For classification tasks, we used fixed task definitions and balanced downstream splits to control class imbalance. Thus, Accuracy is meaningful under the same split/metric protocol across methods. F1 was also included in the original protocol to complement Accuracy under balanced splits, with consistent conclusions, but was omitted from the main tables due to MICCAI space constraints; we apologize for not making this explicit.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper is overall interesting and tackles an important problem at scale, but concerns regarding potential information leakage from metadata conditioning, limited clarity in the experimental protocol, and incomplete baseline comparisons weaken confidence in the results; these issues, along with missing implementation details, should be clarified and can likely be addressed during the rebuttal.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The reviewers agreed that the paper presents an ambitious and technically strong fMRI model with substantial empirical validation, and the rebuttal adequately addressed the main concerns regarding evaluation clarity, metadata conditioning, and experimental protocol, supporting acceptance.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This is a strong study with brain foundation models trained on a large dataset and validated extensively on many downstream tasks. The strengths clearly outweighs the weaknesses.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This paper contributes a strong and timely foundation-model direction for multi-state fMRI analysis through metadata-conditioned pretraining and a broad representational framing that goes beyond a single downstream task. The rebuttal appears to have directly addressed the main protocol-related concerns and clarified the intended scope of the contribution, which likely improved confidence after rebuttal more than for many other papers in the pool. Given the significance of the direction and the reasonably successful rebuttal, this paper merits acceptance.