List of Papers Browse by Subject Areas Author List
Abstract
Multimodal large language models (MLLMs) hold great potential for medicine, as they inherit knowledge from LLM and allow multiple data modalities to be integrated, analysed and interpreted in natural language. However, the field of medical MLLMs is constrained by non-trivial challenges, notably the scarcity of high-quality training data and the frequent occurrence of missing data in the real-world clinical setting. Here, we propose a novel unified multimodal model, UniBrain, for brain magnetic resonance image (MRI) analysis. To address potential missing brain MRI modalities, we employ a unified training strategy to perform joint imaging modality imputation and brain image understanding. During training, an interleaved and description-enriched data flow is constructed to train the model in an autoregressive manner, enabling medical reasoning with generated multimodal data. A self-alignment strategy is introduced to leverage dense image embeddings to learn fine-grained anatomical features without requiring detailed image captions. Furthermore, we propose a dynamic hidden state mechanism to alleviate the exposure bias during long-context multimodal inference. Extensive experiments on multi-disease brain MRI dataset demonstrate that UniBrain achieves high performance for brain image imputation, understanding, and disease diagnosis under various extents of modality incompleteness. Source code is publicly available at https://github.com/zhiyuns/UniBrain.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2626_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/zhiyuns/UniBrain
Link to the Dataset(s)
N/A
BibTex
@InProceedings{SonZhi_Unified_MICCAI2026,
author = { Song, Zhiyun AND Liu, Che AND Xia, Tian AND Kori, Avinash AND Bai, Wenjia},
title = { { Unified Multimodal Model for Brain MRI Imputation and Understanding } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16894},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This work proposes a unified multimodal model for brain MRI analysis with missing modalities. It formulates the tasks of missing modality imputation and final diagnostic understanding as a single autoregressive sequence modeling problem. The work leverages dense visual embeddings of the input images to align the understanding and generation branches, mitigating the lack of fine-grained textual supervision and train the model using self-generated intermediate states, reducing error accumulation in long-context autoregressive reasoning.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1 integrates generation and understanding in a single unified framework, avoiding the disconnect of traditional explicit imputation pipelines 2 SA enhances fine-grained image representation, and DHS improves stability for long-chain reasoning
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1 experiments mainly conducted on 2D slices; 3D consistency remains to be evaluated. A potential concern is the mismatch between the slice-level training setup and the case-level radiology reports. Since the findings/impression are associated with the full MRI examination rather than a batch 2D slices, the textual supervision may be only weakly aligned with slices input. This could introduce supervision noise and makes it unclear whether the reported text-generation and understanding performance truly reflects fine-grained slice-level reasoning. 2 large model size requires multiple GPUs, limiting practical deployment 3 The paper argues that unifying generation and understanding leads to mutual benefits. However, the dedicated imputation baselines used for comparison, such as MM-GAN and ResViT, are relatively dated. It is therefore unclear whether the reported gains would still hold against more recent and stronger medical image synthesis baselines. 4 The role of the self-alignment (SA) loss is not fully clear. Since the flow-matching objective already trains the model to generate missing modalities, it remains unclear what additional capability SA provides beyond being an auxiliary reconstruction objective. The paper would benefit from a clearer explanation of how SA differs from flow loss, and stronger evidence that it indeed improves cross-branch alignment rather than simply acting as an extra regularizer for image reconstruction.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please refer to the major strengths.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper presents UniBrain, a unified method for generation and understanding. The generation tasks are aimed at missing modality imputation, which can aid brain image understanding tasks. The tasks are performed sequentially, generating the missing modalities before making decisions with a self-alignment introduced to avoid the need for textual descriptions at training and the dynamic hidden state mechanism allowing the model to be aware of its own generation biases.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper addresses an important and challenging problem in multimodal learning, handling missing modalities, and demonstrates promising results. The proposed unification of generation and understanding is a novel approach to this issue in medical imaging. By combining these components, the method removes the need for a two-stage training pipeline (i.e., separate generation and downstream models) and enables the model to account for biases introduced during image generation through the proposed dynamic hidden state mechanism. Additionally, integrating all modalities within a unified framework allows for a more comprehensive representation of patient status, better reflecting real-world clinical decision-making.
- The paper is clearly written and easy to follow, with well-designed figures that effectively support and enhance understanding.
- The experimental results are well-aligned with the stated objectives and demonstrate the effectiveness of the proposed UniBrain approach.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Given that the generation task is performed sequentially, it is unclear whether, in cases where multiple modalities are missing, changing the order of generation tasks would affect the final performance.
- In Table 2, the authors evaluate missing modality imputation and state that ensembling results from five random seeds improves performance over fully generative baselines. However, it is unclear how the performance of these generative baselines would change if they were subjected to the same ensembling strategy. Therefore, the claim that “if UniBrain is ensembled with five random seeds, it can outperform specifically designed imputation models” appears to be overstated, as the baselines are not evaluated under comparable conditions.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposes a novel approach that unifies generation and understanding, enabling improved performance in scenarios with missing modalities. The method is both novel and conceptually interesting, and I believe it offers meaningful value to the MICCAI community. The experimental evaluation is thorough and demonstrates the effectiveness of the approach. Overall, the strengths outweigh the weaknesses, and I recommend a weak accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This manuscript proposes UniBrain, a multimodal framework based on BAGEL for brain MRI analysis. Key innovations include self-alignment for fine-grained representations, dynamic hidden states ro reduce exposure bias and a training strategy enabling reasoning with generated images.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This manuscript addresses a clinically relevant problem.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The manuscript lacks details on multi-disease dataset. Out-domain results are required. Quantitative metrics are standard and but missing radiologist evaluation for imputation fidelity, diagnosis agreement or interpretability of generated reasoning. Comparisons are limited to generative baseline, not SOTA medical MLLMs.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
methodological novelty.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We address major comments as summarised by the Area Chair, listed below.
1.The role of the self-alignment loss (R1.4). The self-alignment loss (L_SA) plays an important role in learning image representations and ViT tokens. Although the modality imputation loss (L_flow) already performs reconstruction for the missing modalities, empirical observations showed that using L_flow alone results in significantly slower convergence for the generation branch compared to the understanding branch. This may indicate that the representations learnt by the model, while optimal for diagnosis, lacked the fine-grained features to condition the image generation branch. Therefore, L_SA is introduced to enforce the model to condition its generation on the ViT tokens, rather than exclusively relying on low-level VAE latents. L_SA bridges the representation gap between the two branches. This is quantitatively validated in Table 3, where introducing L_SA improves both image fidelity for generation and Top-1 Acc for understanding.
2.Order of modality generation (R2.1). UniBrain was trained using a randomised order of MRI modalities, which encourages the model to learn permutation-invariant representations. Empirical evaluations show that altering the generation order results in negligible fluctuations in diagnostic accuracy (<2%).
3.2D vs. 3D supervision gap (R1.1). To mitigate potential noise, our framework employs a rigorous data preprocessing pipeline, which was abbreviated in the initial manuscript due to space constraints. First, we utilised the dataset’s lesion masks to selectively sample 2D slices, ensuring they align with the reports. Second, volumetric and size-related descriptions in the original reports (e.g., 76x72x69 mm^3) were parsed and projected into 2D slice dimensional descriptions. Such a pipeline improves the alignment between images and text and mitigates the supervision noise. In addition, extending our framework to natively support 3D volumetric data will be an important direction for future work.
4.Ensembling comparison (R2.2). We wish to clarify that both single-seed (UniBrain) and multi-seed sampled (UniBrain*) results are explicitly reported in Table 2 to ensure strict fairness. When comparing single-seed UniBrain to baseline generative methods, in terms of PSNR, UniBrain only slightly underperforms the baselines that are purely optimised for pixel-level reconstruction task. In terms of diagnostic performance, single-seed UniBrain achieves substantially higher performance (improving Top-1 Accuracy from 61.70% to 74.47% for T1c imputation). This highlights a core advantage of our unified architecture: it prioritizes the synthesis of clinically meaningful, high-frequency contexts over mere perceptual fidelity. The 5-seed UniBrain results, following the convention in many diffusion-based image translation methods, are included to demonstrate that averaging generations can lead to generation metrics (PSNR/SSIM) comparable to strong generative baselines.
5.Baselines and SOTA MLLMs (R1.3, R3.3). We would respectfully direct the reviewers to Table 1 and Table 2, where we benchmarked against several SOTA models. We include SimMLM (ICCV, 2025) iand M2DN (TMI, 2024) in addition to the classic MM-GAN and ResViT methods for IRL and EMI. For medical MLLMs, we compare with UniMedVL and Lingshu, both published in 2025.We believe this justifies the timeliness of comparison results.
6.Limitations and future works (R1.2, R3.1, R3.2). In the revised manuscript, we will add limitations or relevant future works in the Conclusion Section: 1) Computational cost: The requirement for GPU (with over 32G memory) during inference; 2) Generalisation: The current lack of out-of-distribution (OOD) test datasets across diverse disease cohorts; and 3) Clinical validation: The necessity for future radiologist evaluations for imputation fidelity and diagnosis agreement, to supplement existing automated metrics.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper introduces an autoregressive framework that unifies missing modality imputation and diagnostic understanding. Reviewers unanimously recommend acceptance. Instead of a two-stage imputation pipelines, Dynamic Hidden States is used to mitigate exposure bias in long-context autoregressive reasoning. This is an interesting work for the MICCAI community with a clear technical contribution.
While the methodological novelty is high, the authors must address the following technical concerns in the final manuscript:
1.Explicitly differentiate the Self-Alignment (SA) loss from the flow-matching objective (R1). Clarify how SA actively improves cross-branch alignment rather than merely functioning as an auxiliary image reconstruction regularizer.
2.Since generation is sequential, add a discussion/analysis on how the specific ordering of modality generation impacts final performance when multiple modalities are missing simultaneously (R2).
- Clarify the gap between 2D slice-level visual training and 3D case-level radiological text supervision (R1), discussing the potential impact of supervision noise.
4.Ensembling comparison (R2): If needed, correct the claim in Table 2 regarding missing modality imputation. Comparing a 5-seed ensembled UniBrain against single-seed generative baselines may be unfair. Please clarify.
5.The imputation baselines (MM-GAN, ResViT) are not the most recent, adding more up-to-date references is needed in the literature review (R1 R3). Also please justify the absence of comparisons against recent SOTA medical MLLMs.
6.In the discussion section, please explicitly state the computational limitations (multi-GPU requirements for inference) and the lack of out-of-distribution testing or radiologist-graded fidelity evaluations (R1, R3).