List of Papers Browse by Subject Areas Author List
Abstract
Deep learning models deployed in clinical imaging frequently encounter distribution shifts, yet most out‑of‑distribution (OOD) detection methods are evaluated only on controlled research datasets. As a result, it is unclear whether existing approaches can reliably identify segmentation failures that arise in real‑world clinical practice. We evaluated six OOD detection methods on a deployed liver CT segmentation model (3D nnU‑Net) using internal data from 400 patients and external data from 100 patients collected across nearly 70 sites in 7 countries. One method was Pairwise Surface DSC, a surface-based extension of Pairwise DSC, that we introduced. OOD performance was measured using sensitivity, AUROC, and balanced accuracy, with thresholds determined on an independent cohort of 400 patients using the Youden J statistic. Statistical significance was assessed using McNemar tests and stratified bootstraps (α = 0.05) with Benjamini–Hochberg correction.
Pairwise Surface DSC was the top-performing method, with perfect sensitivities (1.00), near-perfect AUROCs (0.97 internal; 1.00 external), and the highest balanced accuracies (0.94 internal; 0.88 external; p<0.001). These results show that automated failure detection for liver CT segmentation is clinically feasible and that Pairwise Surface DSC is a promising candidate for deployment.
Our code is available at https://github.com/mckellwoodland/liver_ct_ood_translation.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/6055_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/mckellwoodland/liver_ct_ood_translation
Link to the Dataset(s)
N/A
BibTex
@InProceedings{BenJed_ARealWorld_MICCAI2026,
author = { Bennett, Jeddy AND Woodland, McKell AND Castelo, Austin AND Altaie, Mais AND Anthony, Ajith AND Siddiqi, Noreen S. AND Long, James P. AND Brock, Kristy K.},
title = { { A Real-World Evaluation of Failure Detection for Liver CT Segmentation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16886},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper compares several techniques for detecting out-of-distribution samples in a liver segmentation model deployed in a real clinical workflow. The proposed pairwise surface DSC metric appears to outperform the other methods evaluated. However, the small number of failure cases in the final test set raises some concern about how strongly the results support the claims.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The introduction of pairwise surface DSC is a clever way to make pairwise DSC more sensitive to segmentation failures, since most failures occur at the object boundary. 2.Evaluating the method on data from a clinically deployed model, collected under routine clinical conditions, is a strong and practical choice. 3.The proposed method clearly outperforms the other OOD techniques considered in this study.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The test set contains only 18 failures, so the positive class is very small. In this setting, it is unclear what additional value stratified bootstrapping with replacement provides. 2.The study focuses only on OOD techniques derived from the final model prediction. It would be stronger to include methods based on encoder features and input metadata as well. 3.The rater used to set the validation thresholds appears more aggressive than the two raters used for test-set evaluation on the Likert scale. This makes the failure definition somewhat ambiguous and may bias the DSC-based methods. 4.It would strengthen the paper to evaluate the method on another modality or task to better support generalizability.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think the suggested technique is a clever and simple method to detect segmentation failures in real world deployments. The effectiveness is supported by the results compared to previous techniques, despite the limitations of the dataset.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
1.The paper addresses automated detection of clinically unacceptable liver CT segmentations from a deployed 3D nnU-Net model. 2.The main contribution is a real-world comparative evaluation of six OOD/failure detection methods, including the proposed Pairwise Surface DSC, which extends Pairwise DSC with a surface-based agreement metric. 3.The paper claims to improve over prior controlled OOD evaluations by testing on routine clinical scans and by using a failure signal that is more aligned with clinical contour acceptability. 4.The evidence presented includes independent threshold selection on 400 validation patients, internal testing on 400 patients, external testing on 100 patients from nearly 70 sites in 7 countries, expert Likert-based review, and formal statistical testing. 5.The practical implication is a deployable safety layer for liver CT segmentation that may help route failed cases for manual review.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper evaluates a clinically important deployment question that is often discussed but rarely tested on real-world data. 2.The study design is strong: thresholds are set on an independent validation cohort, and testing includes both internal and very diverse external data with expert review.  3.The proposed Pairwise Surface DSC is simple, well motivated, and aligned with contour acceptability because it emphasizes surfaces rather than only volumes. 4.Results are impressive: Pairwise Surface DSC achieves perfect sensitivity on both internal and external test sets, with AUROC 0.97 internally and 1.00 externally, and the best balanced accuracy. 5.The paper includes proper confidence intervals, corrected significance testing, and discussion of deployment issues, which strengthens the translational credibility.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The methodological novelty is incremental; Pairwise Surface DSC is an elegant variant of Pairwise DSC, but not a fundamentally new OOD paradigm. 2.The number of failures in the test sets is small (13 internal, 5 external), so sensitivity estimates are still statistically fragile despite looking perfect. 3.The reference standard is clinically meaningful but subjective, and the exact failure definition may vary across institutions and workflows. 4.The paper studies only one organ, one task, and one model class, so generalization to smaller structures or multi-class segmentation is unknown. 5.The strongest methods require multiple ensemble predictions, but computational cost and deployment latency are not discussed. 6.One external-institution scan was unintentionally included in the internal test set because of a database error; likely minor, but it reflects imperfect data provenance.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My score is based on the high importance of safe deployment, the moderate but useful novelty of Pairwise Surface DSC, and the strong evidence from real-world multi-site evaluation. The main positives are the unusually practical study design and the strong comparative performance of the proposed method. The main concerns are the limited number of failures and the incremental nature of the technical novelty.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper proposes a new method for detecting inference failures due to out of distribution data called Pairwise Surface DSC, and applies this to whole liver segmentation from CT images. The technique is similar to a prior method called Pairwise DSC, which calculates the dice score between multiple segmentations which are normally ensembled to produce a single output. The new method swaps out DSC with Surface DSC, which is similar to a Hausdorff distance and is thought to be more relevant to clinical impact. The new method is compared to 5 other OOD detection methods, evaluated on an internal and diverse external dataset, using expert ratings as the reference. The proposed method has generally better performance.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The topic of AI reliability in the clinical setting is very important and relevant to this conference
- This work has strong statistical methods, especially useful for pairwise comparison of methods
- Using a subjective but multi-reader Likert score as the reference is considered a strength, as this can capture a variety of failure modes.
- The diverse external dataset (from 7 countries) is valuable and strengthens generalizability
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The technical innovation relative to Pairwise DSC is small (although impact is notable)
- The clinical need for precise liver segmentation is overstated. It is not used in an automated way which could impact safety today.
- The number of failures in both datasets is small (<5%). This is a good thing in general, but the imbalance makes some of the statistics weaker (sens = 1.0, AUROC = 1.0). I have no suggestions to address this but it has a minor impact on interpreting the results, perhaps meriting discussion.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- It is noted that the code is available, which is a strength. Please clarify the public availability of the datasets used.
- The authors may have indirectly identified themselves in the abstract
- Consider citing something for the Surface DSC metric (https://doi.org/10.1002/acm2.13647)
- The subfigures in the lower row of Fig 2 are a little hard to read, especially with the compressed y-axis. Consider using a single plot with log transforming the abscissa
- Should include tests of significance for age & sex in table 1 to strengthen the claim of distribution shift
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The impact of this work is relative to the prior work with Pairwise DSC. Compared to that paper this work has a thoughtful but small technical change (shifting to surface DSC from DSC), and a strong demonstration in liver CT, with a particularly strong external dataset.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the reviewers for their valuable time and feedback. Reviewers consistently recognized the importance of evaluating OOD detection in a clinical setting and the strength of our large-scale, multi-site study (900 patients, ~70 sites, expert review), as well as the strong performance of the proposed Pairwise Surface DSC (AUROC 0.97 internal, 1.00 external).
Below, we respond to three key points of reviewer feedback: the limited number of test failure cases, the incremental novelty of Pairwise Surface DSC, and the focus on liver segmentation. 1.To address the small number of failures, we reported p-values and their associated statistical power for each performance comparison (Table 4). The low failure rate reflects a key practical challenge in deployed clinical OOD detection systems: manual identification and labeling of failures is time-consuming and impractical at scale as models improve, and failures become rarer. We hope our work will motivate the development of methods to set robust detection thresholds in clinical environments where access to real failure cases is limited. 2.The principal contribution of our study is a robust evaluation of six OOD detection methods on a deployed segmentation model using real-world clinical data. Pairwise Surface DSC, our secondary contribution, represents a practical and effective extension of Pairwise DSC. 3.It is important for future work to extend the evaluation, especially that of Pairwise Surface DSC, to additional imaging modalities and anatomical sites. Such an extension was not feasible within a single study, as rigorous clinical assessment of OOD detection for even one deployed segmentation model required a large amount of resources.
We will now respond to the remaining points of study-related feedback:
- Stratified bootstrapping with replacement was used to provide 95% confidence intervals as uncertainty metrics for each evaluation metric.
- We agree that comparison with feature-based methods would be valuable. However, in our experience (doi: 10.59275/j.melba.2024-g93a), storing feature representations for large training cohorts requires substantial memory. In contrast, training- and output-based methods are lightweight and require fewer hyperparameter choices and distributional assumptions, making them more suitable for real-world clinical systems.
- While failure labeling is inherently subjective, our protocol reflects real clinical practice. The radiologist with seven years of experience applying a more conservative standard than the other two readers highlights realistic inter-rater variability, while majority voting produced robust labels for the test dataset.
- Our methodology adds negligible computational overhead to a standard nnU-Net pipeline, as it reuses model outputs from cross-validation for ensembling.
- Regarding the clinical importance of liver segmentation, the evaluated model is a real clinical system that has been used thousands of times at our institution. For a concrete example of its role in clinical practice, we refer readers to our related liver ablation clinical trial (doi: 10.1016/S2468-1253(25)00024-X).
- The accidental inclusion of a single external scan in the internal cohort reflects a realistic clinical data ingestion error scenario.
- We corrected a missing comma in the abstract that inadvertently implied authorship of Pairwise DSC.
Overall, our work provides one of the most realistic and clinically grounded evaluations of OOD detection for medical image segmentation to date.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper recevied three consistent reviews of accept.