Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Medical AI systems are often evaluated on datasets where ground truth labels suffer from inter-rater variability. Calibration is typically computed against majority-vote consensus to assess whether predicted probabilities reflect correctness. However, recent work argues that calibration to human majority is theoretically flawed in inherently subjective tasks, calling into question the utility of calibration error in such settings. In this work, we demonstrate that this flaw is not with calibration itself but arises from label aggregation strategies, such as majority voting, that induce a mismatch between the calibration target and the intended meaning of the model predictions. Specifically, we show that this mismatch breaks the oracle guarantee that a predictor returning the true probabilities should achieve zero calibration error in expectation. We then consider two principled alternatives to eliminate this mismatch: (i) computing calibration directly against individual rater labels, and (ii) retaining consensus labels while analytically aligning predictions with the aggregation rule. We prove that both approaches recover consistent calibration estimates that restore the oracle guarantee, showing that calibration remains meaningful under label stochasticity. Through controlled sensitivity analyses and a real-world breast density case study, we empirically validate the theoretical behavior of the two protocols. Together, this work establishes the first principled framework that demonstrates the oracle guarantee of calibration error under subjective labeling.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4416_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/whiterabbit-ai/calibration-under-rater-disagreement

Link to the Dataset(s)

N/A

BibTex

@InProceedings{KumHar_On_MICCAI2026,
        author = { Kumar, Harshit AND Truong Vu, Yen Nhi AND Su, Jason AND Matthews, Thomas Paul},
        title = { { On the Behavior of Calibration Error Under Rater Disagreement } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16895},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The paper addresses a meaningful and important problem: traditional ECE is not suitable in multi-rater settings. This touches on a fundamental issue—how to properly model and interpret the relationship between multiple annotations and the underlying ground truth. The proposed methods are relatively comprehensive and take multiple factors into consideration.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well-organized, concise, and rigorous. The authors clearly present their arguments and provide effective validation that supports their claims. The benchmark experiments and various figures are strong and effectively support the claims.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The introduction of AdjustedECE appears somewhat ad hoc. The theoretical justification and experimental analysis could be further strengthened. For example, it would be helpful to evaluate the stability of the metric under varying numbers of predictions or raters, or to explore whether the metric can provide guidance for model optimization.

    Since the purpose of a metric is to evaluate model performance, the paper could benefit from additional comparative experiments across different models. Including other evaluation metrics would help better demonstrate the reliability and practical value of the proposed methods.

    Treating multi-rater annotations as a single label via majority voting is clearly suboptimal. In fact, in multi-annotator segmentation tasks (especially pixel-wise settings), this issue has already been widely recognized, and related research is relatively mature. Conceptually, ideas like MR-ECE and AdjustedECE may already be implicitly present in such works, which raises some concerns about the novelty. On the other hand, this work may still provide useful validation of these ideas in such settings.

    Regarding the “Oracle Guarantee,” if achieving this property is claimed as a key contribution, it would be important to further consider practical factors such as prediction uncertainty and annotation noise. A deeper theoretical and empirical analysis would strengthen the overall contribution.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper is well-written and the methodology is clearly presented. The research problem is important and relevant. However, the theoretical analysis and empirical validation lack sufficient rigor, which limits the credibility and the overall strength of the contribution.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The work discusses the behaviour of Expected Calibration Errors as a metric to quantify multi-rater agreements with ground truth labels aggregated with majority voting. The authors then formalise the majority voting as a binomial distribution, which inhrenently induces non-linear transformations of the latent single-rating probability, consequently breaking the oracle guarantee of ECE in cases where the empirical positive fraction and bin-average prediction do not converge at higher n. The authors discuss two alternative corrective ECE-based metrics, namely MR-ECE evaluated against individual raters, and AdjustedECE, which also applies majority voting to the model prediction. A simple theoretical exploration suggests that these two metrics accurately model the oracle prediction cases. Synthetic experiments and a simple real-world breast density task validated these thereotical claims.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The points made by the authors ( majority voting non-linear transformation of multi-rater ground truths leading to erroneous calibration ) are well substantiated by the intuitive, easy-to-follow proofs.
    • The work further adds on meaningful points and discussions to previous works on calibration metrics, especially the work by Baan et al, and instigates on the limitations of established metrics.
    • The work is highly accessible; it highlights that two alternative, relatively simple modifications of ECEs can already lead to better alignment between predictions and multi-rater ground truths
    • The sensitivity analysis of the synthetic voting situation covers a broad scenarios of voter subjectivity/objectivity, the results align to the points addressed by the authors.
    • The breast density case is well chosen and highlights the real-world applicability to the points made by the author.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • While the authors acknowledge the strong i.i.d rater assumption in the conclusion, a deeper analysis of the theoretical framework in real multi-rater cases ( violation of i.i.d from systematic biases and raters knowledge levels ) would strengthen the framework.
    • The practical limitation of AdjustedECE ( pushing prediction towards extremes in the breast density task ) is worth discussing more. Particularly, more in details of the specific strengths and weaknesses for MR-ECE and AdjustedECE, and which one would be a better choice for differing cases.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper raises an interesting point on the relationship between calibration error and multi-rater agreement. The authors substantiate their point with intuitive theoretical proofs and results. Reviewing is also a multi-rater, subjective voting, and I (subjectively) feel that the paper, in contrast to the multitude of works attempting to get the highest metric scores in certain tasks, initiates a refreshing discussion as to why we, in the machine learning community, shouldn’t just blindly use any well-established metric ( “just because others have used it in the past” ).

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper investigates the behavior of Expected Calibration Error (ECE) in medical imaging tasks characterized by inter-rater disagreement, formalizing the problem through a latent probability model. The authors theoretically demonstrate that computing ECE against a majority-vote consensus label (CaseECE) inherently violates the “oracle guarantee,” as the majority voting mechanism applies a non-linear transformation to the true latent uncertainties. To restore theoretical validity, the work evaluates two alternative protocols: it provides a mathematical justification for MR-ECE, an existing method that computes calibration directly against individual rater annotations, and introduces a novel metric called AdjustedECE, which analytically aligns model predictions with the consensus aggregation transformation. Finally, these theoretical claims are empirically supported through a synthetic benchmark controlling for task subjectivity and a real-world binary classification case study on breast density involving five radiologists.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Theoretical Formalization of a Clinical Problem: A significant strength of this work is its theoretical framing of an important real-world clinical issue. The authors establish a latent probability model to formalize the problem of calibration under inter-rater disagreement. By recognizing that multi-rater annotations are often reduced to a single majority-vote label, the authors mathematically identify that this aggregation induces a non-linear transformation on the true latent probabilities. This provides a clear, principled explanation for why standard calibration error against consensus labels (CaseECE) is theoretically flawed and violates the Oracle Guarantee. This formalization moves the discussion of label subjectivity from an empirical observation to a grounded theoretical problem. Principled Framework and Analytical Adjustments: Rather than simply abandoning the use of expected calibration error (ECE) in subjective settings, as some recent work has suggested, the authors provide a principled framework to restore its validity. They offer mathematical proofs that two alternative protocols—MR-ECE (evaluating against individual rater labels) and AdjustedECE (aligning predictions with the non-linear consensus transformation)—asymptotically recover consistent calibration estimates and preserve the Oracle Guarantee. The AdjustedECE metric, in particular, offers an analytically sound way to evaluate model calibration when only aggregated consensus labels are available. Controlled Synthetic Benchmark for Subjectivity: The authors introduce a controlled synthetic benchmark to systematically validate their theoretical findings. By using a concentration parameter ($a$) to generate bimodal probability distributions, they create synthetic worlds (A-F) that span different degrees of intrinsic task subjectivity—from near-deterministic labeling to highly subjective cases. This allows for a rigorous and interpretable sensitivity analysis, clearly demonstrating that MR-ECE and AdjustedECE maintain the Oracle Guarantee across varying levels of simulated inter-rater disagreement, whereas CaseECE consistently deviates. Demonstration of Real-World Implications: The paper effectively bridges theory and practice through a real-world case study on breast density classification. By evaluating a model against labels provided by five radiologists, the authors empirically demonstrate the phenomenon they theorized: standard CaseECE suggests systematic miscalibration, whereas the proposed protocols (MR-ECE and AdjustedECE) reveal that the model is actually well-calibrated. This section strongly highlights the practical and regulatory implications of their findings, showing that the choice of evaluation metric under label stochasticity can materially alter conclusions regarding model safety and performance.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Overstated Novelty and Marginal Algorithmic Contribution: The paper claims to establish the “first principled framework” for subjective label calibration, yet acknowledges that its primary baseline protocol, MR-ECE, already exists in recent literature. Consequently, the actual methodological novelty is restricted solely to the introduction of AdjustedECE and its associated theoretical proofs, rendering the algorithmic contribution too incremental for a top-tier venue.

    2.Fundamentally Flawed Clinical Assumption (i.i.d. Raters): The entire mathematical foundation of the consensus transformation and the oracle guarantee relies strictly on the assumption that raters are independent and identically distributed. In clinical reality, annotators exhibit heterogeneous diagnostic thresholds and frequently make correlated errors due to shared image artifacts, meaning that violating this unaddressed assumption would cause the theoretical foundation of AdjustedECE to completely collapse.

    3.Practical Limitations of AdjustedECE: Even if the theoretical assumptions hold, AdjustedECE suffers from severe practical flaws, notably its reliance on a fixed number of raters per case, which is rarely true in real-world clinical datasets. Furthermore, its non-linear transformation aggressively pushes probability scores toward the extremes (0 and 1), hollowing out the mid-probability range and severely degrading the visual informativeness of reliability diagrams, thereby defeating the primary clinical purpose of calibration evaluation.

    4.Insufficient and “Oracle-Biased” Empirical Validation: The empirical validation is exceptionally weak, relying almost exclusively on the “oracle” setting without stress-testing the metrics against realistically misspecified (e.g., systematically over/underconfident) models. Moreover, the real-world validation is limited to a single, opaque, proprietary binary classification dataset with a fixed number of 5 raters, completely lacking evaluation on more complex tasks like multi-class classification or lesion segmentation where rater disagreement is paramount.

    5.Lack of Statistical Rigor: The reported ECE values are presented merely as point estimates without any quantification of statistical significance, confidence intervals, or variance across raters. Without evaluating the sensitivity of the metrics to binning artifacts or reporting bootstrap resamples, it is impossible to rigorously determine whether the performance differences observed in the case study are statistically meaningful or simply artifacts of sample variability.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My recommendation of a Weak Reject is primarily based on the gap between the paper’s theoretical framework and the complexities of real-world clinical settings, alongside the need for more comprehensive empirical validation.

    While the authors address a highly relevant and important clinical challenge—calibration under rater disagreement—the core theoretical foundation of the proposed AdjustedECE relies heavily on the assumption that raters act as independent and identically distributed variables. In standard clinical practice, this assumption is often difficult to satisfy, as annotators frequently exhibit correlated judgments and varying diagnostic thresholds. Discussing or relaxing this assumption would be crucial for real-world applicability. Furthermore, the non-linear transformation introduced by AdjustedECE tends to push probability scores toward the extremes. While theoretically sound, this behavior could practically limit the informative value of clinical reliability diagrams, which are essential for visual calibration assessment.

    Finally, the empirical validation, though well-intentioned, is currently limited to a single proprietary dataset with a fixed panel of five raters. To meet the high standards of MICCAI, the manuscript would benefit significantly from broader testing on more diverse, complex tasks (e.g., multi-class or segmentation) and from stress-testing the metrics against realistically misspecified models. Addressing these limitations would greatly strengthen the manuscript’s contribution and its readiness for publication.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    Although I appreciate the authors’ efforts to address the statistical significance via bootstrap analysis, I maintain my decision of Weak Reject. The rebuttal explicitly acknowledges that relaxing the idealized i.i.d. assumption for realistic, heterogeneous raters would require a fundamentally different transformation (a Poisson-binomial h_n), effectively admitting that the proposed g_n formulation is clinically misaligned and relegating the actual real-world solution to “future work.” Furthermore, the authors concede that AdjustedECE severely degrades the visual resolution of reliability diagrams, compromising its core practical utility. Coupled with the continued reliance on a single proprietary dataset and the absence of validation on complex clinical tasks (e.g., multi-class or segmentation), the manuscript’s framework remains too disconnected from real-world clinical applications to meet the standards of a top-tier venue.



Author Feedback

We thank the reviewers for their feedback and recognizing the paper’s clinical relevance, theoretical grounding, and clarity.

[Novelty, R1/3] Our work highlights an important phenomenon: label aggregation can change what it means to be calibrated. Aggregation strategies induce transformations on the label distribution, and non-identity transformations (e.g., g_n in case of majority vote) create a mismatch between model output and calibration target. Thus, calibration error should be measured against a well-defined calibration target. This differs from prior work: [1] designs MR-ECE for increased stability; [2,3,4,5] use soft-overlap metrics and per-rater ECE to retain information about rater disagreement. Our theory reveals a symmetry in the calibration equation: the mismatch can be corrected either on the label side (MR-ECE) or the prediction side (AdjustedECE). We thus provide another justification for MR-ECE and introduce a novel metric AdjustedECE which naturally arises from the aggregation transformation.

[Justification, R1/3] Though softer metrics have been studied in segmentation, many medical datasets obtain labels through majority voting [6,7]. Majority voting under i.i.d. raters provides a mathematically clean setting for isolating aggregation-induced target mismatch. Further, the model already accounts for R3’s concern: p_j for case j includes the impact of image content, such as artifacts. The empirical findings from the breast density study align with the theory, supporting the simplified model. Our presentation focuses on accessibility and clarity, laying the foundation for future work, including extension to complex annotation rules and other tasks, such as multi-class and segmentation settings.

[Generalization, R1/2/3] If aggregation strategy changes or i.i.d. assumption is relaxed, the form of both g_n and AdjustedECE will change. E.g., for heterogeneous raters i with probabilities p_j^i, majority voting induces a Poisson-binomial h_n instead of g_n; CaseECE still creates target mismatch; MR-ECE shifts target from p_j to mean of p_j^i over i, and AdjustedECE applies h_n instead of g_n, restoring the Oracle Guarantee.

[MR-ECE vs AdjustedECE, R1/2/3] We will add a table to compare MR-ECE vs AdjustedECE: no aggregation assumption vs. assumed aggregation map; requires rater-level labels vs. only consensus labels needed; higher reliability-diagram resolution vs. lower resolution with growing number of raters. Thus, MR-ECE is preferable when rater-level labels are available, while AdjustedECE is useful for medical datasets that report only consensus labels [6,7]. In practice, the impact of the lower resolution of AdjustedECE can be limited by the small number of raters per exam in medical datasets and be further mitigated by common binning strategies [8].

[Stability, R1/3] Fig. 3 already varies rater count (5–500); we will further vary sample/bin. In the most subjective world (a=1.0), with 10 bins/5 raters, increasing samples 50->500->5000 gives CaseECE 0.158->0.106->0.094, which plateaus above zero, while MR-ECE 0.070->0.020->0.004 and AdjustedECE 0.094->0.058->0.011 approach the oracle value. With 1000 samples/5 raters, increasing bins 5->15->50 gives CaseECE 0.086->0.099->0.111, MR-ECE 0.011->0.015->0.028, and AdjustedECE 0.020->0.031->0.053.Thus, MR-ECE/AdjustedECE show finite-sample/binning effects but remain closer to the oracle value than CaseECE.

[Statistical significance, R3] Over 10,000 bootstraps, CaseECE > individual-rater ECE for 4/5 raters, and significant for 3/5 raters (p<0.005). Two exceptions were a non-significant difference (p=0.080) and a rater with higher individual-rater ECE. Bootstrap 95% CIs and one-sided paired tests will be provided.

[1] MR-ECE,UNSURE,2026 [2] Multi-Rater Agreement Modeling,CVPR,2021 [3] CalDiff,IEEE JBHI,2026 [4] CURVAS,Comput. Biol. Med.,2025 [5] TwinTrack,MIDL,2026 [6] VinDr-CXR,Sci. Data,2022 [7] CheXpert,AAAI,2019 [8] Measuring Calibration in DL,CVPR,2019



Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper investigates the limitation of Expected Calibration Error (ECE) as a metric to quantify multi-rater agreements with ground truth labels from majority voting. As alternatives, besides MR-ECE that is evaluated against individual raters, the paper proposes the AdjustedECE metric. The theoretical claims are validated with synthetic experiments and a real-world breast density task.

    The work offers interesting insights into the relationship between calibration error and multi-rater agreement, which is meaningful for various applications. The experiments and the proofs are well done and well support the claims. However, the reviewers also raised a few key points to be addressed: 1.The theoretical justification/discussion and experimental analysis of the proposed AdjustedECE could be further strengthened (e.g., metric stability, application-specific trade-off); 2.Robustness of the i.i.d. assumption for different raters need to be verified/discussed; 3.The novelty of the AdjustedECE can be better elaborated in the context of the prior works.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The rebuttal has addressed the reviewers’ concerns on novelty, i.i.d assumption (with potential extension), and method justification. Although additional validation for more complex applications would be appreciated, the work itself addresses an important issue and can offer some interesting discussion for the community.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Reject

  • Please justify your recommendation.

    The paper is interesting and adresses an important challenge in multi-rater assessment. However, its strong reliance on what could be non practical assumptions identified by two reviewers limits its potential clinical impact and places it marginally under the acceptance bar



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper received mixed reviews. All reviewers appreciate the theoretical analysis to formalize the relationship between calibration error and multi-rater agreement. The reviewers have concerns mainly about experimental validation. Although the proposed AdjustedECE is not strictly advantageous in the considered settings over the existing MR-ECE, the key contribution is in designing an alternative possibility of generalizing ECE to multi-rater scenarios. The paper can generate interest in the community for future exploration in this direction.



back to top