List of Papers Browse by Subject Areas Author List
Abstract
Differences in imaging modalities across institutions (e.g., MRI and CT) introduce significant challenges in deploying federated learning in real-world settings for medical image segmentation. To address this challenge, we propose FedMoSA, a robust federated framework for handling severe modality heterogeneity, where clients may be fully modality-disjoint. FedMoSA explicitly decouples modality-specific and modality-agnostic representation learning through the integration of specialized adapters and shared adapters within the image encoder of SAM(Segment Anything Model). To enable stable optimization under modality heterogeneity, we introduce modality-restricted aggregation for specialized adapters and modality-balanced aggregation for shared components. Furthermore, we incorporate a prototype-based alignment mechanism to promote consistent shared adapter representations across clients. We evaluate FedMoSA on publicly available multi-institution, multi-modal liver segmentation datasets under diverse federated configurations, including explicit generalizability testing. FedMoSA achieves up to +8.5 Dice improvement under fully modality-disjoint setting over state-of-the-art federated medical image segmentation methods, with a substantial reduction in inter-client performance variance. The code is available at https://github.com/obedjamir/FedMoSA.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0744_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/obedjamir/FedMoSA
Link to the Dataset(s)
N/A
BibTex
@InProceedings{JamObe_Federated_MICCAI2026,
author = { Jamir, Obed AND Paul, Angshuman},
title = { { Federated Medical Image Segmentation under Modality Heterogeneity via Specialized Adapters } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16881},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a framework for medical image segmentation under modality heterogeneity. Building on SAM, they design modality-specific adapters to capture modality-specific features, while a shared adapter helps the model focus on general features and prevents it from drifting away from the federation. A loss function, along with server-side aggregation of modality-specific and shared parameters, is proposed, resulting in a generalizable model for scenarios with extreme modality heterogeneity.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The targeted problem is timely, and the overall presentation of the method is smooth. They propose a modality-agnostic approach that enables organ segmentation under modality heterogeneity, which can arise in many practical real-world situations.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
There are unclear points mainly in the implementation and evaluations. These points are as follows:
1- The proposed method is only evaluated on liver segmentation, as it can be considered a relatively simple task where training a smaller model than SAM on limited data can still provide reasonable performance. Evaluating the method on more challenging tasks is needed.
2- In this work, the SAM decoder as well as the shared and modality-specific adapters are trained, which can result in a very long training time compared to nnUNet and this is not reflected here. Also, selecting 25 percent of the slices per 3D case at each epoch and running the training for 25 rounds might lead to convergence for liver segmentation, would that still be the case for more challenging tasks?
3- There are enough publicly available CT and MRI datasets available, why isn’t the model tested on real-world client data distributions? Details on the clients’ data distribution in each modality, as well as the number and modality distribution of the test sets, are lacking. Is the metric averaged on both modalities?
4- In Protocol 3, where clients are fully modality-disjoint, how does FednnUNet handle federated aggregation among all clients? Are federated models trained specifically for each modality? There are some details missing in the implementation.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method and application are sound and timely, but there are some concerns regarding the implementation and the fair evaluation of the method (mentioned above) that require clarification.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal addresses several of my main concerns by clarifying the evaluation scope, providing additional evidence beyond liver segmentation, explaining the training and communication cost, and resolving key implementation ambiguities. These clarifications make the experimental setup and conclusions more convincing than in the original submission. My remaining concerns regarding moderate novelty and generalizability still stand, but given the limited scope for revisions and the strengths of the work, I believe the paper can proceed. However, these clarifications should be reflected in the camera-ready version.
Review #2
- Please describe the contribution of the paper
The paper proposes FedMoSA, a federated learning framework designed for medical image segmentation under modality heterogeneity, where different clients may have data from different imaging modalities (e.g., MRI vs CT), including fully disjoint settings. The method extends SAM-based architectures by introducing modality-specialized adapters to capture modality-specific features and shared adapters to learn modality-agnostic representations. To support this design, the paper proposes modality-restricted aggregation (aggregating specialized adapters only across clients with the same modality) and modality-balanced aggregation (weighting shared components based on both dataset size and modality diversity), along with a prototype alignment loss to enforce consistency in shared representations across clients. Experiments on multi-modal liver segmentation datasets under varying heterogeneity settings show that the method improves robustness and performance, particularly in fully modality-disjoint scenarios where standard federated approaches degrade
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper addresses federated medical image segmentation under modality heterogeneity, including fully modality-disjoint settings. It proposes a framework that separates modality-specific and shared representations using specialized and shared adapters within a SAM-based model. The aggregation is adapted to this setting through modality-restricted aggregation for specialized adapters and modality-balanced aggregation for shared components, which aims to reduce cross-modality interference. A prototype alignment loss is introduced to encourage consistent shared representations across clients. The evaluation covers multiple datasets and heterogeneity protocols, including challenging fully disjoint scenarios, and includes ablation studies to assess the contribution of each component, as well as qualitative and representation analyses supporting the design choices
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper has several limitations. The proposed use of adapters within a SAM-based architecture builds directly on prior work such as MSA (Medical SAM Adapter) and related adapter-based fine-tuning approaches, and the idea of separating shared and specialized components is conceptually similar to existing personalized federated learning methods that partition parameters into global and local parts. As such, the architectural contribution appears to be an extension of existing designs rather than a fundamentally new formulation. The aggregation strategies, including modality-restricted and modality-balanced aggregation, are intuitive but resemble grouped or weighted aggregation schemes commonly used in federated learning, and the novelty relative to prior approaches is not fully clarified. The prototype alignment mechanism is also related to prior work on prototype-based or representation alignment methods (e.g., prototypical networks and FL approaches using class or feature prototypes), and its contribution beyond existing alignment techniques is somewhat incremental.
In terms of evaluation, experiments are limited to liver segmentation datasets, which may restrict the generality of the conclusions to other organs, tasks, or imaging settings. Although multiple datasets are used, they are combined into a specific benchmark, and it is unclear how the method performs in more diverse or real-world multi-institution scenarios. Finally, while an out-of-distribution experiment is included, it is relatively limited (few rounds of fine-tuning), and does not fully demonstrate robustness to unseen modalities or larger domain shifts.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses a relevant and practical problem (modality heterogeneity) with a reasonably well-motivated design and solid empirical evaluation, particularly under challenging fully modality-disjoint settings. While some components build on existing ideas (adapters, prototype alignment, weighted aggregation), the combination is coherent and shows consistent improvements. The work is sound overall, though the level of novelty is moderate and certain assumptions (e.g., known modalities) and evaluation scope could be further strengthened.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper presents FedMoSA, which extends MSA so that MSA can be trained on clients with inconsistent imaging modalities, such as MRI and CT. To stabilize training, the authors add specialized aggregation and a prototype mechanism to push shared adapters to learn modality-agnostic representations.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This seems to be the first work on multimodal FL with SAM. The highlights are modality-specific/shared adapter design and the use of proximal regularization to align the shared adapters.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main weakness is in the evaluation. The authors did not include baseline aggregation strategies specifically designed to cope with multimodality/data heterogeneity, such as:
[1] K. Borazjani, N. Khosravan, L. Ying and S. Hosseinalipour, “Multi-Modal Federated Learning for Cancer Staging Over Non-IID Datasets With Unbalanced Modalities,” in IEEE Transactions on Medical Imaging, vol. 44, no. 1, pp. 556-573, Jan. 2025, doi: 10.1109/TMI.2024.3450855.[2] Fan Qi, Shuai Li; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 12312-12321
and more classical ones such as FedProx, etc. Instead, the only comparison is to FedAvg.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the evaluation is not as thorough as I expected, the idea and task here provide valuable insight for the community. I also like the open source aspect of this.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal adequately addresses my main concern about baselines. Plus, the problem is timely, the benchmark is non-trivial, the contribution is real, and the code is open. Hence, I recommend acceptance.
Author Feedback
1.Evaluation Scope and Generalizability (R1-Q1, R1-Q2, R1-Q3, R2, META-Q2): We appreciate the reviewers’ suggestion to evaluate more challenging tasks. We selected liver segmentation because it offers one of the most diverse collections of publicly available, multi-institutional, multimodal datasets (six datasets spanning four CT and two MRI sources) that reflect real-world client data distributions. The collapse of FednnU-Net (AFA) to 61.07% Dice and FedMSA to 83.01% Dice under Protocol 3 (Table 1) demonstrates that the benchmark is non-trivial for modality-heterogeneous FL. To further assess generalizability, as shown in Fig. 5, we evaluated FedMoSA on BraTS for brain tumor segmentation across T2 and FLAIR under a fully modality-disjoint setting. FedMoSA achieved 88.76 Dice with only five fine-tuning rounds (META-Q2), suggesting the federated optimization is not anatomy-specific and addressing R1-Q3 on convergence. Evaluating additional organs remains a valuable future direction.
2.On the Novelty (R2, META-Q1): While some components of our design are known in the literature, our primary contribution is their principled combination for modality-disjoint federated learning (FL), a setting in which cross-modality aggregation can cause measurable performance collapse. For example, FedMSA, which uses the same SAM backbone but lacks modality-specialized adapters, drops to 72.24% IoU under Protocol 3, while FedMoSA achieves 84.85% (+12.6 IoU). This improvement disappears under Protocol 1, where heterogeneity is minimal, confirming that the observed gain is specific to modality-disjoint conditions. Furthermore, prototype-based alignment at the shared adapter feature level has not been explored in prior FL work; this approach reduces inter-client variance by 15x (9.8e-7 vs. MSA’s 1.5e-5, Fig. 4c). Our modality-balanced weighting strategy jointly accounts for both dataset size and modality diversity, a combination not present in previous FL aggregation methods. Finally, ablation studies in Fig. 4a and 4b demonstrate that replacing either aggregation strategy with FedAvg consistently degrades performance.
3.Missing Baselines (R3, META-Q3): We thank the reviewers for these references. Borazjani et al. focus on multi-modal fusion for cancer staging, combining multiple distinct data types per patient. In contrast, our setting uses only medical images, each with a single modality (CT or MRI), and clients are fully modality-disjoint. The model must generalize across modalities without access to paired data, making direct comparison infeasible. Qi & Li’s hypergraph framework assumes a public multi-modal dataset on the server, which violates our privacy constraints. FedProx, a regularization method, does not address modality-specific aggregation but will be included as an additional baseline in the revision.
4.Implementation Details (R1-Q2, R1-Q3, R1-Q4, META-Q3): Training cost (R1-Q2): Only lightweight adapters (bottleneck dim 48) and the decoder are updated and communicated each round, reducing per-round communication to 35.96 MB for two modalities. This represents an 87.3% reduction compared to the 378 MB full model (Table 2). Data distribution (R1-Q3): Section 3 details the six datasets, 80:10:10 patient-level splits, and three protocol configurations. The global test set consists of equal, non-overlapping samples from all clients. FednnU-Net under Protocol 3 (R1-Q4): FFE aggregates all parameters via FedAvg without modality awareness, while AFA aggregates only structurally identical components. Local metrics are reported per client, and global metrics reflect joint CT and MRI performance.
5.Known Modality Assumption (R2): We appreciate the reviewer’s concern. In clinical practice, modality labels are available and routinely used. Our assumption of known modality matches real-world conditions. Studying dynamic routing for unseen modalities is a valuable future direction, as we note in our conclusion.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a SAM-based efficient federated segmentation framework for modality-heterogeneous medical data. The topic is relevant and practical, as modality heterogeneity is a common challenge in real-world federated medical imaging settings. Based on the reviewers’ comments and my own assessment, I recommend that this paper proceed to the rebuttal stage. The authors should address several key issues: first, the method appears relatively simple, with aggregation weights mainly determined by modality availability and sample size, while local prototype learning is a common strategy for improving federated consistency; the authors should clarify the distinctiveness and advantages of their design. Second, the method should be evaluated on more challenging organ segmentation datasets to better demonstrate its robustness and generalizability. Third, the experimental details should be expanded, and recent works on federated domain shift or heterogeneity mitigation should be included for comparison to ensure a fair and up-to-date assessment.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper addresses an important and practical problem in federated medical image segmentation, namely modality heterogeneity across clients, including fully modality-disjoint settings. The proposed FedMoSA framework is well motivated: separating shared and modality-specific adapters within a SAM-based model, together with modality-aware aggregation and prototype alignment, provides a coherent solution to reduce harmful cross-modality interference. The initial reviews raised valid concerns about the moderate novelty of the components, the evaluation being centered mainly on liver segmentation, and the lack of comparisons with more specialized heterogeneity-aware FL baselines. The rebuttal addresses these concerns reasonably well. In particular, the authors clarify the distinction from standard adapter-based SAM and generic prototype alignment, provide evidence that the gains are most pronounced under modality-disjoint settings, add a BraTS-based evaluation to support generalizability beyond liver segmentation, and explain why some proposed baselines are not directly comparable under their privacy and modality assumptions. The additional implementation details on patient-level splits, communication cost, and protocol design also make the experimental setting clearer. Some limitations remain, especially the moderate methodological novelty and the need for broader validation across more organs and unseen modalities. Still, the problem setting is timely, the benchmark is non-trivial, the method is technically sound, and the empirical improvements under severe modality heterogeneity are convincing. I therefore support acceptance, with the expectation that the final version includes the clarified baseline discussion and additional experimental details from the rebuttal.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After reviewing the authors’ rebuttal and the post-rebuttal recommendations from reviewers, I recommend Accept. The rebuttal effectively addresses the key concerns.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes FedMoSA, a SAM-based federated learning framework for medical image segmentation under modality heterogeneity. Based on the reviewers’ feedback, the authors’ rebuttal addresses the main concerns, and I recommend accepting this manuscript.
