Paper Info Reviews Author Feedback Meta-Review Back to top

List of Papers Browse by Subject Areas Author List

Abstract

Federated Learning (FL) has recently emerged as a popular paradigm in healthcare, enabling collaborative model training across hospitals while preserving patient privacy. However, distribution shifts arising from different acquisition devices, patient populations, or clinical protocols can lead to substantial performance degradation when models are deployed at new sites. Out-of-distribution (OOD) performance estimation is thus critical to assess this degradation, but labelled OOD data from target hospitals is scarce or expensive to obtain. Agreement-on-the-Line (AotL) (Baek et al., 2022) addresses this by predicting OOD accuracy without labelled data via agreement between pairs of model checkpoints, though obtaining multiple models for this purpose is computationally expensive. We observe that FL naturally resolves this: diverse client checkpoints are already produced during training at no additional cost. We propose FedAgree, a method to facilitate agreement-based OOD evaluation in federated settings by leveraging both local and cross-client checkpoints. We introduce five checkpoint strategies that progressively expand the use of cross-client information and evaluate them on diverse medical imaging modalities (dermoscopy, retinopathy, histopathology), under both IID and non-IID settings. Our empirical results demonstrate that federated healthcare settings offer an ideal environment for practical, label-free OOD evaluation.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5130_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/MLO-lab/FedAgree

Link to the Dataset(s)

CIFAR10.1: https://github.com/modestyachts/CIFAR-10.1 CIFAR10.2: https://github.com/modestyachts/cifar-10.2 HAM10000 (DermaMNIST): https://medmnist.com/ BCN20000: https://www.kaggle.com/datasets/pasutchien/bcn20000?select=bcn_20k_train.csv APTOS: http://kaggle.com/datasets/sovitrath/diabetic-retinopathy-224x224-2019-data DeepDRiD (RetinaMNIST): https://medmnist.com/ Camelyon17-WILDS: https://wilds.stanford.edu/datasets/#camelyon17

BibTex

@InProceedings{SerGiu_FedAgree_MICCAI2026,
        author = { Serra, Giuseppe AND Werner, Ben AND Buettner, Florian},
        title = { { FedAgree: Label-Free Performance Estimation Under Distribution Shift for Federated Medical Imaging Analysis } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16880},
        month = {September},
        page = {pending}
}

Reviews

Review #1

Please describe the contribution of the paper

The paper proposes FedAgree, a method to facilitate agreement-based OOD evaluation in federated settings by leveraging both local and cross-client checkpoints. It introduces five distinct checkpoints strategy (local anchor, anchor peer, history, peer and full) that expand the use of cross-client information evaluating them on different medical imaging modalities under both IID and non-IID settings. The empirical results demonstrate that federated healthcare networks provide an ideal enviroment for practical, label-free OOD evaluation.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The core observation is practically valuable: FL training naturally produces diverse model checkpoints at no additional cost, and exploiting them for OOD performance estimation is a well-motivated and non-obvious insight. The connection to AotL is principled and clearly formalized.
- The non-IID evaluation on Camelyon17-WILDS is convincing and clinically relevant: each client maps to a different hospital, which directly reflects realistic FL deployment. The improvements over all baselines are substantial (e. g. , FedAgree-Full achieves MAE of 14.80 vs 34.67 for AotL-Local).
- The R² coefficient is proposed as a practical reliability check for the linear fit, which is a useful operational tool for practitioners to decide when to trust the OOD estimate.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- In the IID setting with FedAvg, all clients receive the identical global model at the start of each round and train on data drawn from the same distribution. Client checkpoints within the same round therefore differ only due to local mini-batch stochasticity, resulting in nearly identical models across clients. This undermines the core motivation for cross-client checkpoint diversity in Table 2, and the observed gains may not reflect genuine multi-client benefit. The paper does not discuss this confound.
- HAM10000 was collected from multiple clinical sites with different acquisition modalities. The paper treats it as an IID dataset with uniform partitioning, ignoring the site metadata. A non-IID split based on acquisition source would have been a clinically motivated setup directly analogous to Camelyon17, and would have produced a more meaningful evaluation. This is a missed opportunity that weakens the medical FL claim.
- K=4 clients is too small to support general conclusions. With only 4 clients, checkpoint diversity is limited even in non-IID settings. It is unclear how FedAgree scales in larger federations (K=10, 20, 50) where diversity may increase but variance may also grow. This is not discussed in the paper nor in limitations.
- Low R² in non-IID settings is not operationally addressed. The paper reports R² values of 0.30–0.72 on Camelyon17 and notes that estimates should be treated with caution, but provides no guidance on what practitioners should do when R² is low in a real clinical deployment. For a paper claiming clinical utility, this is a significant gap.
- Density estimation baselines are excluded without justification. Methods based on density estimation share the same objective as FedAgree (estimating OOD performance without labels) and are cited in the related work. Their exclusion from the comparison is not explained. While they may require access to raw target data — which conflicts with FL privacy constraints — this should be stated explicitly as the reason for exclusion.
- Fig. 1 is not self-explanatory. It uses four overlapping symbol types representing different quantities in the same plot. Understanding what each symbol means requires simultaneously reading the caption and Section 3.2.For the only figure in the paper, which should communicate the central intuition, this is a clarity issue.
- Privacy implications of checkpoint sharing are not discussed. The paper assumes trusted partners under institutional agreements, but sharing all checkpoints from all rounds exposes the full training trajectory of each client. This is a non-trivial privacy concern in medical FL that deserves at least a brief discussion.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

The authors provide a link to an anonymized repository, but most files in the repository are empty. This effectively prevents reproducibility of the reported results. The authors are strongly encouraged to ensure the repository is complete before final submission.
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

FedAgree addresses a practically relevant problem and the idea of exploiting FL checkpoints for OOD performance estimation is well-motivated. The non-IID results on Camelyon17 are genuinely convincing and demonstrate a clear benefit. However, the IID experiments — which constitute the majority of Table 2 — are methodologically confounded by the use of FedAvg with IID partitioning, where cross-client checkpoints are nearly identical and provide no meaningful diversity. Furthermore, HAM10000’s multi-source structure is ignored in favor of an artificial IID split, missing an obvious and clinically motivated non-IID evaluation opportunity. Combined with the small federation size (K=4), the lack of operational guidance for low-R² scenarios, and the absence of justification for excluding density estimation baselines, the experimental evidence is insufficient to support the generality of the claims.
Reviewer confidence

Very confident (4)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

N/A
[Post rebuttal] Please justify your final decision from above.

N/A

Review #2

Please describe the contribution of the paper

This article aims to address the challenge of OOD generalization in federated learning frameworks based on agreement-on-the-line. While AotL is typically computationally expensive, the core claim states that federated learning naturally addresses this issue, simplifying the process of OOD generalization via model agreement.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed method, FedAgree, is a very simple and natural extension of Agreement-on-the-Line for the FL framework. The intuition that federated learning naturally produces diverse checkpoints across clients at no extra training cost is clear and convincing.

This article includes an exhaustive analysis of the impact of 6 different checkpoint strategies, which strengthens the empirical soundness of the evaluation.

The authors provide anonymized code.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

Storing all model checkpoints for each client at each round is inefficient with respect to space. In low-resource environments, it may not be feasible to save so much data for evaluation. Have the authors considered any techniques to mitigate the space requirement? I would appreciate a discussion of the space complexity in the final manuscript.

Figure 1 is not referenced, nor discussed in the article

The experiments lack ablation. While the analysis of various checkpoint strategies can, in a sense, be considered an ablation on its own, the article would be strengthened by further analysis. For example, how does the performance change as the number of clients increases/decreases? How does it change as more training rounds are conducted? Do checkpoints from early rounds (when the models are not yet well trained) carry useful information relative to ones collected from later rounds? I am also curious if aggregation strategies other than FedAvg have a significant impact on the performance. What about varying levels of heterogeneity in data volume/class-balancing across clients? I understand that rebuttal space is limited and the authors may not be able to touch on all of these points, but I would certainly appreciate a bit more insight into some of these topics.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

N/A
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I appreciate the simplicity of the approach, and that the results sufficiently demonstrate the effectiveness of FedAgree across multiple checkpoint strategies. However, I have concerns about the overall space complexity of the approach, particularly in lower resource environments. I also find it the experimental analysis somewhat underwhelming. The lack of ablation of number of clients, training time, etc raise some additional concerns for me. That said, I would be willing to increase my score given a strong rebuttal on these points as I do find the method to be overall interesting and relevant to the MICCAI community.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

The authors adequately address my concerns about space complexity and certain aspects of ablation analysis. I am happy to accept this paper.

Review #3

Please describe the contribution of the paper

The authors evaluate the OOD (‘out of distribution’) of datasets trained at different ‘sites’, a method to evaluate different training distributions. The authors perform this with training data from various datasets, such as CIFAR and HAM1000.

The method is solid for evaluating federated learning distributions between sites. The sites have similar enough distributions that it’s difficult to state how far this could be pushed; however the concept is demonstrated and it makes sense as an evaluation during FL training.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This was performed on a variety of datasets and demonstrated a clear bias between sites, which was able to be quantitated.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

The agreements of the OOD distributions were very similar distributions, which is a good metric. It would be useful to know how far the metrics could have been pushed for OOD datasets.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The authors claimed to release the source code and/or dataset upon acceptance of the submission.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

N/A
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a solid method and the authors demonstrated it clearly. It may not be a game changer in terms of FL methods, but is useful to characterise.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

Similar to above, the results are fine, just not very exciting. The authors justified their work, addressing reviewer comments within the limitations provided.

Author Feedback

We thank all reviewers for their constructive feedback. Reviewers consistently acknowledged that FedAgree addresses a practically relevant problem, that exploiting FL checkpoints for OOD estimation is a well-motivated and non-obvious idea, and that the method is simple, effective across multiple checkpoint strategies, and relevant to the MICCAI community. We address the main concerns below.

Nr of clients: K=4 represents a lower-bound for FedAgree: being the regression fit based on pairwise agreement points, fewer clients = fewer regression points = noisier line fit; K=10/20/50 would strictly increase the pool of variance-reduced agreement pairs. Consistent with this, History/Peer/Full, which enlarge the checkpoint pool temporally, outperform Local Anchor in Table 2, confirming that more checkpoints lead to lower MAE. Similarly, larger values of T could increase the reliability of the method.

IID diversity: We respectfully disagree with R1’s statement that cross-client diversity in the IID setting reduces to mini-batch stochasticity. Two independent sources of randomness shape each client’s model: SGD stochasticity, and the variance from the local dataset itself being a finite IID draw of the shared distribution. The latter makes FedAgree’s pairwise agreements more informative due to the induced variance. The non-trivial gaps between AotL (which uses models differing only by mini-batch stochasticity) and FedAgree variants in Table 2 empirically confirm this.

HAM10000 Non-IID experiments: HAM10000 contains images from only two clinical sites (Vienna and Melbourne). A non-IID split by acquisition source would therefore yield K=2, which we believe does not reflect a realistic federated setting and has been therefore not considered.

Missing baselines: Density estimation methods target OOD detection rather than OOD accuracy estimation, and require access to raw target data features, making them inapplicable under FL privacy constraints. We will clarify this in the related work.

Storage complexity: For K=4, T=10, and a ResNet18 (~42.7 MB), total storage per client is ~427 MB — feasible in clinical federated infrastructures and advantageous compared to storing high-resolution medical imaging data. In extreme resource-constrained edge devices, FedAgree can operate via streaming peer checkpoint pairs on demand within the secured partner network, discarding them immediately — reducing working memory to 2x ~42.7 MB at any time.

Time complexity: The (regression + prediction) phase adds minimal overhead. On Camelyon (the largest dataset), intermediate strategies require only ~0.8s on average (~1.5× AotL); the Full strategy ~2.3s (~4.0× AotL), which remains a negligible overhead anyway.

Early- vs late-round checkpoints: Early-round checkpoints, although coming from not well-trained models, are valuable because they help ‘draw’ the agreement line. Using only very similar checkpoints produce clustered agreement points that provide little information about the line’s slope and intercept. Early checkpoints introduce models from distinct regions of the loss landscape, which contribute to a better-spread set of points and a more robust linear fit.

Low-R^2 scenarios: FedAgree provides an uncertainty-aware performance estimation approach with R^2 being a practical reliability check. As a practical rule, if R^2 is low in safety-critical scenarios, we recommend falling back to manual performance estimation by labeling a representative set of images as a secure alternative.

Client heterogeneity: Our experiment on Camelyon17 already provides a setting with heterogeneous data volumes across clients (H1: ~59K, H2: ~35K, H3: ~85K, H4: ~130K, H5: ~147K). We acknowledge that further exploring client heterogeneity is an interesting aspect left for future work.

Code: The anonymized repository linked in the paper is complete and visualisation of empty files may have been a temporary issue with the anonymized repository service.

Meta-Review

Meta-review #1

Your recommendation

Invite for Rebuttal
Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a label-free method for estimating OOD performance by leveraging model checkpoints from both local and cross-client training. The method builds on Agreement-on-the-Line and introduces five checkpoint selection strategies. The framework is evaluated on multiple datasets under both IID and non-IID settings.

The reviews are mixed. This paper addresses an important and practical problem, the idea is clear and well-motivated, and empirical evaluation is sound. However, there are also several concerns need to be carefully addressed: 1.The IID FedAvg experiments may be confounded and may not support the claimed benefit. 2.The experimental analysis lacks certain aspects. 3.Storage and privacy costs of checkpoint sharing are not adequately discussed. 4.Baseline selection and exclusions need further justification.
After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

The paper proposes FedAgree, a method that adapts Agreement-on-the-Line to federated learning for label-free out-of-distribution performance estimation. Based on the post-rebuttal positions, I recommend Accept. The rebuttal evidently resolved R2’s concerns about space complexity and ablation analysis, and R3 found the authors’ clarifications are adequate.

Meta-review #2

After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

The paper proposes an interesting methodology for OOD accuracy estimation based on agreement on the line (AotL) concept. It appears to be a sufficiently novel application of the AotL concept and the results are fairly convincing. Most of the concerns raised by the reviewers have been clarified in the rebuttal. Hence, the paper can be accepted.

Meta-review #3

After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

Two reviewers recommend to accept this paper post-rebuttal. One reviewer did not submit a post-rebuttal assessment (pre-rebuttal: WR).

The authors submitted an effective rebuttal clarifying raised concerns such as choice of clients K, stochasticity of IID sample draws, time/storage complexity.

The idea and approach to evaluate OOD performance in an federated learning environment is well motivated and, even though methodologically rather simple, pretty effective.

back to top

FedAgree: Label-Free Performance Estimation Under Distribution Shift for Federated Medical Imaging Analysis

Author(s):