List of Papers Browse by Subject Areas Author List
Abstract
Manually refining radiological segmentation masks is highly resource-intensive. To determine when this expert commitment is truly justified for the training of segmentation models, we investigate the relationship between label quality and model performance. Expanding beyond models trained directly for inference, we conduct the first study isolating the impact of label quality in pre-training datasets. While high-quality labels remain essential for models proceeding directly to deployment, we find no evidence that strict label quality is crucial for pre-training efficacy. These results question the necessity of exhaustive human-in-the-loop refinement for massive corpora intended for pretraining and suggest that expert effort is more effectively invested in well-curated downstream target datasets.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3404_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
https://amos22.grand-challenge.org/
https://github.com/HiLab-git/WORD
https://github.com/JunMa11/AbdomenCT-1K
https://github.com/MrGiovanni/AbdomenAtlas
https://flare.grand-challenge.org/Data/
https://codalab.lisn.upsaclay.fr/competitions/843
BibTex
@InProceedings{JauAle_Good_MICCAI2026,
author = { Jaus, Alexander AND Marinov, Zdravko AND Reiß, Simon AND Seibold, Constantin AND Wei, Jiale AND Kleesiek, Jens AND Stiefelhagen, Rainer},
title = { { Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper makes a data-centric contribution by systematically investigating how label quality affects medical image segmentation, particularly distinguishing between direct training (in-domain use) and pre-training + fine-tuning scenarios. Instead of proposing a new model, it introduces a large-scale, controlled experimental framework where high-quality annotations are replaced with realistic pseudo-labels generated by different models (e. g. , nnU-Net, MedSAM, TotalSegmentator), thereby simulating varying levels of annotation noise. Using this setup across multiple datasets and segmentation models, the paper reveals a key finding: while label quality is crucial for models trained and deployed directly, it is surprisingly less important for pre-training, as fine-tuning on clean data largely compensates for noisy pre-training labels. Model performance strongly correlates with label quality in the in-domain setting, but this dependency largely disappears after fine-tuning. Based on these findings, the paper provides actionable insights for dataset construction—suggesting that expensive expert refinement may not be necessary for large-scale pre-training datasets, and that effort should instead focus on high-quality downstream annotations.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
First, it addresses a highly relevant and practical problem in medical imaging: the trade-off between annotation quality and scalability. Given that large datasets such as DAP-Atlas or AbdomenAtlas require thousands of hours of expert effort, the paper tackles an important question—whether such costly refinement is necessary—making the work highly impactful for both dataset construction and model development. Second, the paper proposes a novel and well-designed experimental framework to study label quality. Instead of relying on synthetic noise (as commonly done in prior works using perturbations or morphological operations), it generates realistic pseudo-label noise using a diverse set of models (nnU-Net, TotalSegmentator, STU-Net variants, MedSAM). This setup closely mimics modern dataset curation pipelines and avoids unrealistic assumptions about noise distributions, which is a clear methodological improvement over prior studies
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
First, the study relies entirely on pseudo-labels generated by models as a proxy for annotation noise, which may not fully capture real-world annotation errors. Although this is more realistic than synthetic perturbations, it still differs from genuine human annotation variability. As a result, the conclusions may not fully generalize to real clinical annotation settings. Second, the experimental setup, while controlled, is somewhat limited in model diversity. The study fixes a single segmentation architecture (DynUNet / nnU-Net style training) to isolate data effects. While this improves internal validity, it raises concerns about external validity: it is unclear whether the conclusions hold for other architectures, especially newer foundation models or transformer-based segmentation approaches.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend a Weak Accept (4/6). The main factor supporting this score is the paper’s strong problem formulation and practical relevance. It addresses an important and timely question in medical imaging—whether high-quality annotations are necessary at scale—which has clear implications for dataset construction and resource allocation. The study is particularly valuable because it reflects realistic modern data pipelines based on pseudo-labeling, rather than relying on artificial noise. Another key strength is the novel experimental perspective, especially the explicit separation between in-domain training and pre-training + fine-tuning. The finding that label quality is critical for direct deployment but largely irrelevant for pre-training is both non-trivial and impactful, and could influence how large-scale medical datasets are built. The experimental design is also carefully controlled and comprehensive, covering multiple datasets, label generators, and evaluation settings, which makes the conclusions convincing.
However, the score is limited to weak accept due to several factors. The novelty is moderate, as the general topic of label noise and annotation quality has been studied before, and the paper mainly extends prior work to a larger-scale and pre-training setting. In addition, the study relies on pseudo-labels as a proxy for annotation noise, which may not fully reflect real-world clinical annotation variability. The work is also largely empirical, with limited theoretical or mechanistic analysis of why the observed phenomena occur.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This study provides a pragmatic framework for optimizing annotation efforts in medical image segmentation by revealing a fundamental asymmetry between pre-training and finetuning stages. The authors demonstrate that while downstream performance is highly sensitive to the precision of fine-tuning segmentation labels, large-scale pre-training on cohorts such as Word, Amos, and CT1K-Abd is remarkably robust to noise. By utilizing a diverse range of label generators including nnU-Net, MedSAM, and the STU-Net family to produce masks with varying Dice scores (70-90%), the research proves that suboptimal pseudo-labels can facilitate feature transfer nearly as effectively as expert-curated ones. Given the high cost and limited availability of medical experts, the paper suggests developers prioritize expert labor for refining small-scale fine-tuning sets while leveraging automated, high-throughput labeling for massive pre-training collections.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Extensive and diverse 3D datasets: The study is grounded in a robust experimental framework, utilizing a wide array of large-scale 3D medical segmentation datasets that accurately reflect the complexities of clinical volumetric imaging across various anatomical regions. Comprehensive simulation of label noise: A significant merit of the work is the inclusion of diverse label-generation models, such as nnU-Net, MedSAM, and the STU-Net family. By leveraging these varied architectures to produce a spectrum of pseudo-label qualities, the authors successfully simulate real-world annotation noise and its impact on model training. Clear and coherent presentation: The manuscript is well-structured and the results are presented with a high degree of transparency. The figures and tables are clearly plotted, making the core findings regarding the relationship between pre-training and fine-tuning easy to understand at a glance.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Lack of self-supervised learning baselines: A critical methodological gap is the absence of comparisons with self-supervised learning (SSL) paradigms (e.g., DINO). Without an SSL baseline, it remains unclear whether the pre-training gains stem from the “noisy labels” or simply from the model’s exposure to a larger data volume. Undefined boundary of label noise: While the study shows pre-training is robust to label quality (Dice 70-90%), it fails to explore the “break-down point” of this robustness. It remains unknown whether a noise threshold exists or if even completely randomly generated masks would yield similar results, which leaves the actual necessity of using “pseudo-labels” for pre-training unproven. Technical error in visualization: There is a noticeable technical oversight in Figure 3.c, where the bar representing the STU-Net Large results for the AbdomenAtlas dataset is missing. It is unclear whether this is a plotting error or if the performance value is lower than the visible y-axis limit.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper delivers a large scale empirical study on the role of annotation quality in 3D medical image segmentation. The authors convincingly show that pre training is robust to noisy labels, while fine tuning remains highly sensitive to precision. This insight offers practical guidance for resource allocation, suggesting that automated labeling can be used effectively for pre training, with expert annotation reserved for fine tuning where accuracy is most critical. Despite its strong experimental scale and clear engineering value, the work leaves methodological gaps. The absence of self supervised learning baselines limits the ability to disentangle the effects of label guidance from data volume, and the robustness limits for label quality are not fully explored. Overall, the study provides valuable clarity on label quality requirements in medical pre training, and I recommend a Weak Accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper systematically studies the relationship between label quality and model performance in medical image segmentation. The authors construct pseudo labels using seven segmentation models and quantify pseudo-label quality on four base datasets using Dice and Surface Dice. Based on these, the paper conducts in-domain experiments with DynUNet, and further performs pre-training and fine-tuning experiments on two additional datasets. The results suggest that, in the in-domain setting, model performance is overall positively correlated with dataset quality. In the pre-training experiments, even when relatively low-quality pseudo labels are used for pre-training, the fine-tuned models on the additional datasets still outperform models trained from scratch.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1 The paper addresses an interesting and practically relevant problem, and the use of model-generated pseudo labels to mimic modern dataset construction pipelines is a meaningful and moderately novel experimental setting. 2 The overall logic of the paper is clear, and the experimental setup is described in sufficient detail, supporting reproducibility. 3 The experiments are fairly comprehensive, covering seven pseudo-label generation models and four base datasets, with additional evaluation on one extra model and two extra datasets. 4 The empirical analysis is relatively thorough, and the figures and tables are well organized and easy to follow.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1 I remain unconvinced by the second conclusion in 3.1.The reported performance trend is obtained by fitting a quadratic polynomial to only seven discrete data points, yet several actual nnUNet points deviate noticeably from the fitted curve. From the figure, the performance of Orig does not appear to exceed nnUNet and even drops substantially in some cases. Therefore, the statement “As the dataset quality approaches 100, the performance curves begin to plateau” does not seem to be fully supported by the raw plotted results, and may be overly influenced by the fitted trend rather than directly supported by the discrete experimental observations. 2 In 2.2, the second finding states that “nnUNet outperforms on most datasets, achieving Dice scores from 82.8% to 95.2%.” However, I could not find any value corresponding to “82.8” in the table. This indicates that there is an issue in either the main text or the table, and the reported numbers should be carefully checked and made consistent.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper studies a practically meaningful question, namely how label quality affects model training and transfer performance. Overall, the experiments are fairly comprehensive and the results are clearly presented. However, the current version still contains inconsistencies between the main text and the reported tables/figures, and some conclusions do not fully align with the raw plotted results. This suggests that the interpretation may rely too heavily on the fitted trends while underemphasizing the actual discrete observations.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers for their constructive feedback and their general positive assessment. We address the main concerns below.
Pseudo-labels as a noise proxy (R1). Our pseudo-label setup is intentionally faithful to current large-scale practice, where datasets [8, 20, 27] are themselves constructed from model-generated masks with sparse expert verification, rather than via the synthetic perturbations used in prior work [1, 3, 24]. We acknowledge that for smaller, manually annotated cohorts, inter-rater variability may be the more realistic noise source. For this study, we deliberately keep the noise model consistent across our four datasets, which span 100 to 5,200 cases, to enable a controlled comparison of how the same noise type interacts with dataset scale. Architectural diversity (R1). With well over 100 full 3D segmentation training runs already required to span our 8 label variants, 4 base datasets, and 2 fine-tuning targets, adding an architecture family scales this beyond our available hardware resources. We deliberately fixed DynUnet to isolate the data effect; DynUnet/nnU-Net remains the most consistently competitive 3D backbone in the recent literature, so we believe studying this architecture is the most beneficial to the community. Extending the architecture selection is a natural and interesting follow-up.
SSL baselines and breakdown point (R2). We restricted ourselves to supervised pre-training to avoid a task shift between pre-training and fine-tuning, and because Li et al. [13] (ICLR 2024) report supervised pre-training to be much more sample-efficient than SSL on 3D medical data, so the question for us became how important data quality is under the more efficient supervised pre-training regime. Regarding the breakdown point: MedSAM on AbdomenAtlas reaches only 39.4% Dice and 24.8% Surface Dice (Table 1), well below any plausible expert-curation threshold, yet models pretrained on it still play in the same league with models pretrained on higher quality datasets (Fig.3c). This datapoint already probes the low-quality regime, though a fully randomized-mask control would push it further, and we agree it is an interesting follow-up.
Missing STU-Net Large bar in Fig. 3c (R2). This is not a plotting oversight: STU-Net Large pre-training on AbdomenAtlas did not converge under our fixed, deterministic training schedule. We will mark this explicitly in the camera-ready version and thank R2 for catching it.
Plateau claim and quadratic fit (R3). R3 is right that the 2nd-degree fit smooths the discrete observations. The intended claim that marginal gains shrink at high label quality is most clearly carried by AbdomenAtlas (Fig. 2, right), where the four STU-Net variants span ~4 Dice points in label quality but yield largely overlapping model performance and even perform similarly to the model trained on the original labels.
82.8% inconsistency (R3). Correct: 82.8% is a leftover from BTCV experiments that were removed for spatial reasons from the final manuscript; the actual lower bound in Table 1 is now 83.7% (Word). We thank R3 for catching this and will fix it.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This work investigates how label quality affects medical image segmentation. All reviewers tend to accept it. However, there are also several concerns raised, eg. limited theoretical analysis etc. Please try to fix them if possible.