Abstract

Ensemble disagreement is widely used as a proxy for epistemic uncertainty in medical image segmentation. In practice, many studies form ensembles via K-fold cross-validation (CV), yet refer to them as “deep ensembles” (DE). Because CV members are trained on different data subsets, their disagreement mixes seed-driven variability with data-exposure effects, which can change how uncertainty should be interpreted. We audit recent segmentation uncertainty studies and find that terminology–implementation mismatches are common. We then compare a standard 5-fold CV ensemble to a 5-member DE (fixed training set, different random seeds) under otherwise identical configurations on three multi-rater segmentation datasets spanning three modalities. We evaluate uncertainty for calibration, failure detection, ambiguity modeling, and robustness under distribution shift. DE match segmentation accuracy while improving calibration and failure detection, whereas CV ensembles sometimes correlate more strongly with inter-rater variability on the studied datasets. Thus, ensemble construction should be chosen to match the research question: DE for reliability-oriented use (e.g., selective referral/failure detection) and CV ensembles as a proxy for ambiguity. We provide a lightweight nnU-Net modification enabling DE training within the default pipeline; code will be released upon acceptance.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2355_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/Kirscher/LostInFolds

Link to the Dataset(s)

GleasonXAI dataset : https://springernature.figshare.com/articles/dataset/Pathologist-like_explainable_AI_for_interpretable_Gleason_grading_in_prostate_cancer/27301845 RIGA dataset : https://deepblue.lib.umich.edu/data/concern/data_sets/3b591905z CURVAS dataset : https://zenodo.org/records/13767408

BibTex

@InProceedings{KirTri_Lost_MICCAI2026,
        author = { Kirscher, Tristan AND Bujotzek, Markus AND Kirchhoff, Yannick AND Rokuss, Maximilian AND Isensee, Fabian AND Kahl, Kim-Celine AND Kovacs, Balint AND Maier-Hein, Klaus},
        title = { { Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16882},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    A recent trend is to use cross-validation (CV) folds for uncertainty quantification (UQ), under the name of “deep ensembling” (DE). This work examines this practice, emphasizing the need to improve terminology-implementation mismatches in related literature.

    The principal contributions of this work are (1) highlighting the terminology mismatch, (2) comparing CV and DE for UQ across 3 segmentation datasets, and (3) claiming to provide (upon acceptance) an ensembling extension of nnU-Net.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Using CV for UQ is becoming a common practice for medical segmentation models due to 5-fold CV being the default for nnU-Net; as such, this paper will be of great interest within the MICCAI UQ community. 2.The work brings to light the need of related literature to not call a method “deep ensembling” if each model in the ensemble was trained on different data subsets of an original dataset. Again, this is becoming a relatively common practice due to nnU-Net’s use of 5-fold CV. 3.The paper claims that it will release a nnU-Net modification enabling DE training, which would be of great benefit to the community. 4.The paper compares CV- to DE-based UQ on 3 segmentation datasets (all different modalities) across multiple UQ tasks (calibration, failure detection, ambiguity, modeling, domain shift robustness). 5.Public datasets are used, enabling reproducibility. 6.The utilized datasets release individual segmentations for multiple raters, allowing for an ambiguity analysis. 7.The work uses 95% bootstrapped confidence intervals to demonstrate the variability of the results. It also uses hypothesis testing to evaluate the significance of the results. Furthermore, it has a figure to demonstrate the performance on individual datapoints, so the reader doesn’t just have to rely on point estimates.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The manuscript highlights a niche problem in the literature. While this is useful, the novelty is more incremental than what is typically expected for a full methodological or application paper. 2.The manuscript lacks a grounding in clinical context; the introduction, recommendations, considerations are mostly related to a research setting. The introduction should have briefly summarized why segmentation and uncertainty quantification is important for clinical settings. Furthermore, the discussion/conclusion should have discussed what the findings mean for those developing UQ for segmentation models in a clinical setting. For example, I would have liked to have seen the following question considered: is CV-based calibration/failure detection enough for resource limited clinical settings, i. e. settings in which running 25 nnU-Net submodels at inference is infeasible? 3.The manuscript does not fully explore the distribution shift results, thereby potentially misleading the reader in the final conclusions. For the RIGA dataset, CV performed better than DE on the task of failure detection on scans from a held-out institution. Even so, the manuscript’s main failure detection recommendations are to use DE, without mentioning that this recommendation may not generalize to distribution shifts. Furthermore, while I understand that the authors may have chosen to report effect size for OOD data instead of raw metric values for space considerations, it leaves the reader wondering how the models actually did under distribution shift. 4.Lacks important implementation details, such as the train/test split ratio and if these splits were performed patient-wise (if applicable for the specific public datasets). The lack of knowing how many images is in the test set is especially problematic for the GoldAtlas dataset, as its sample size is small (n=19) and 4 were already withheld for OOD testing. The other important missing detail is the type of hypothesis testing used to get the p-value (i. e. a ** sided ** test).

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    1.To my knowledge, reference [18] does not use CV as a proxy for DE. The sentence implying such should be updated. 2.It would be helpful to define BA-ECE before first use. 3.The term “DE” should not be treated as plural. “Key distinction” should read “DE aims” (not “DE aim”) and “Discussion/Conclusion” should read “DE also incurs” (not “incur”).

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The manuscript highlights a need in related literature to shore up ensembling terminology and compares CV- to DE-based UQ for the task of segmentation, contributions that would be of interest to the UQ community within MICCAI. Furthermore, it is well-written and formulated, evaluating a variety of UQ tasks on multiple public datasets and using statistical testing.

    The reason it was not rated higher is because the contributions are more limited in scope than those from full methodological/application papers. In the rebuttal, I would like to see the requested missing information/discussion added.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    “Lost in the Folds” is a submission that investigates the difference between pure Deep Ensembles, where the only variation consists in the random seeds, and k-fold Cross Validation (CV) model ensembles, which are exposed to different subsets of the data. The manuscript first analyzes the frequency of the confusion between the terms in existing literature, and links the prevalence of CV to the standard choice in the popular nnU-Net framework. Differences between the two approaches are then assessed on three multi-rater medical segmentation datasets spanning MRI (CURVAS) and CT (RIGA) in 3D and retinal fundus imaging (GoldAtlas) in 2D. The aspects that are considered are segmentation accuracy (Dice score), calibration (ACE, BA-ECE), ambiguity (NCC, GED), and failures (AURC). Experimental results provide evidence that predictions of segmentation accuracy, calibration, and failures are best reflected by deep ensembles, while ambiguity correlates better with cross-validation discrepancies. These findings are formulated as practical recommendations.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper highlights a subtle but relevant distinction between two methods that may easily be mixed up in practice, provides very interesting findings, and turns them into actionable items. This includes a small but immediately useful contribution to a widely used library.

    The main definitions and motivation are compelling and clearly stated. The experimental investigation leverages a smart setup with multi-rater datasets and considers three aggregation methods.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The main weakness of the submission in its current state is lack of clarity in the evaluation, which partly undermines confidence in the results.

    First, the paragraph on “Uncertainty quantification and Evaluation” uses notation that is not explicitly declared. It may be guessed that $c$ refers to a specific class, and possibly $v$ to a voxel. $y_n^(v)$ might be the segmentation from the $n$-th annotator in voxel $v$, as $y_n^$ is explained (later!), but then $n$ runs over annotators, while its upper limit $N$ was used for the dataset (with index $i$) at the start of section 3.The way the bins $B_m$ are chosen is not explained, and $v\in B_m$ suggests these bins are voxel sets. The use of the accuracy in a bin that is defined inline is not specified. A short definition for each metric in this paragraph would be very helpful (whereas current eqs. (1) and (2) earlier could be inline). The acronym for the area under the risk curve (AURC) should be given upon first mention. Finally, it would be good to have an explicit statement about how averages at the voxel or image level are taken.

    Second, the statistical tests and OOD comparison in the caption of table 3 are very scarce in terms of details. In order to perform the paired bootstrap test, are the metrics computed by aggregating the voxels at the sample level, and then aggregating the (bootstrapped) samples? Are the significance tests just comparing the confidence intervals with zero? The different way of reporting ID and OOD results, with raw scores in one case and effect sizes in the other, is definitely a cause for confusion. While I agree that in an ideal world one would want to see both (6 lines per metric), if space is an issue, consistently reporting $\Delta$ (2 lines per metric) or both values (4 lines per metric) would be helpful. By comparison, multiplication by 100 does not need to be highlighted with a full sentence in bold, a percent sign in the header line of the table is sufficient.

    Additionally, the literature search that resulted in table 1 is not described in depth, which makes it hard to assess how systematic it was. While this is a high-level weakness, it is not crucial to the paper’s message about the confusion between DE and CV, which would hold even with episodic evidence.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • “Rare structures, underrepresented pathologies, or uneven fold composition” only partially specify the mechanism that relate CV to ambiguities. In fact, for ambiguous samples at inference time, it is likely that similar training examples have different annotations, and CV models exposed to only some of them will give different estimations. Not making this connection explicitly feels like a missed opportunity.
    • Figure 1 seems hard to interpret with respect to a scatter plot that illustrates the same result. Also, even if I understand the desire to show some details in constrained space, two figures out of six metrics end up not being very informative. Please consider the space budget between figures and tables 3-4 as a whole.
    • Technically, the notation of eqs. (1) and (2) indicates parameter sets with DE and CV with the same symbols, as e.g. $\theta_1$ could refer both to $m=1$ or $k=1$. This could be easily solved with a superscript DE/CV. Likewise, the inline expressions for $p$ are not differentiated, and the different dependence on the dataset $\mathcal{D}$ is not explained. In section 4, the index $m=1,\dots,M$ is then used for any ensemble, whether DE or CV.
    • At the very beginning of section 3, it may be helpful to specify the split that the dataset $\mathcal{D}$ corresponds to (development = train + valid?).
    • It would be good to know if excluding the data point from CURVAS in table 2 is standard or a specific choice in this work.
    • Minus signs in tables should be entered in math mode, else they appear as n-dashes which are very short compared to plus signs.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper makes a simple but relevant point which can help the community take informed decisions, and bring awareness to implicit choices. It features very interesting technical details that may be inspiring for future explorations. Although the methods were somewhat hard to decipher from descriptions, the guesses that fill the gaps in experiment description are not big jumps, so I am reasonably confident that results are correct.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper clarifies the essential differences between cross-validation ensembles (CV) and deep ensembles (DE) in uncertainty estimation for medical image segmentation, investigates the misuse of terminology in recent literature, conducts comparative experiments on three multi-rater datasets, provides recommendations for ensemble usage in different scenarios, and modifies nnU-Net to enable support for standard deep ensembles.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The issues identified are highly accurate, pointing out the integration misuse problems that have long been overlooked in the field. The experimental design is rigorous, with only the integration method varied while all other factors kept consistent, leading to reliable conclusions. The provided usage recommendations are highly valuable for engineering practitioners. The paper is well-structured and presents a relatively comprehensive analysis.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The motivation is not fully elaborated, and it is unclear why nnU-Net must have deep integration and the existing integration methods are insufficient. This manuscript consists entirely of experimental analyses and lacks theoretical derivations, such as the approximation relationship between the two and the uncertainty boundary, which are not mentioned. There is insufficient methodological innovation; it is more like a benchmark test and problem clarification without novel algorithm design.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    It is recommended that the author supplement some theoretical analysis to fully explain the necessity of conducting in-depth integration.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses practical issues in uncertainty estimation for medical image segmentation, with rigorous experiments and conclusions that offer reference value to the community. However, it lacks theoretical depth and presents limited methodological innovation; overall, it is marginally acceptable. The authors are required to respond to issues regarding theory and motivation in their revision.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

N/A




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper looks into the practice in medical image segmentation of using cross-validation folds as deep ensembles for uncertainty quantification, which is usual given the prevalence of nnU-Net and the fact that, in that framework, it is standard to train 5 models with different data folds. The reviewers highlight the relevance of the topic to the MICCAI community, the rigorous experimental setup across multi-rater datasets and UQ tasks, and the practical contribution of extending nnU-Net to support standard deep ensembles. Some concerns are raised around clarity in the evaluation section, the depth of the OOD analysis, and missing implementation details, but these are addressable and do not undermine the core message of the paper.



back to top