Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Multiple Instance Learning (MIL) is a standard paradigm for Whole-Slide Image (WSI) analysis and has achieved strong results in computational pathology. However, most MIL pipelines assume a single “gold” label per slide, which conflicts with clinical practice where substantial inter-pathologist variability is common. Existing multi-annotator learning and label-refinement methods typically estimate global annotator reliability or rely on single-instance assumptions, making them poorly suited to MIL and to localized diagnostic contexts where experts disagree. We propose RaLMPH (Reliability-aware Learning for Multi-Pathologist Harmonization), a MIL-based label reconciliation framework for WSIs annotated by multiple pathologists. RaLMPH introduces a reliability field that jointly models (i) local neighborhood structure in WSI feature space and (ii) expert uncertainty (entropy), enabling per-sample identification of trustworthy reference neighborhoods. Leveraging this field, RaLMPH performs sample-wise local annotator ranking to select reliable opinions per slide and applies an adaptive gating mechanism to fuse labels conditioned on local reliability. Experiments on a clinical WSI dataset with labels from six pathologists, as well as controlled simulated benchmarks, show that RaLMPH consistently outperforms existing approaches. Further analyses clarify how our reliability-aware mechanism improves label reconciliation and downstream MIL performance.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1120_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

BRACS: https://www.bracs.icar.cnr.it/ BCNB: https://bcnb.grand-challenge.org/

BibTex

@InProceedings{HonSun_RaLMPH_MICCAI2026,
        author = { Hong, Sungrae AND Jeong, Jiwon AND Cheon, Soeun AND Han, Donghee AND Lee, Sol AND Shin, Jisu AND Kim, Kyungeun AND Yi, Mun Yong},
        title = { { RaLMPH: Reliability-Aware Learning for Multi-Pathologist Harmonization in Whole-Slide Image Classification } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16891},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The authors propose RaLMPH, a Multiple Instance Learning framework for Whole Slide Image classification under multi-annotator disagreement. The core idea is to model the local feature-space neighborhood jointly with annotator-specific trust estimates in order to derive reliable pseudo-labels for ambiguous samples. These labels are then used to supervise two alternative MIL backbones. The method is evaluated on an in-house dataset annotated by six pathology experts at the WSI level, and on a public dataset with synthetically generated multi-rater labels. Competitive performance is reported against several baselines from the label-noise and crowdsourcing literature.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    • Clinically relevant dataset and problem formulation. The use of a real-world WSI dataset with labels from 6 independent pathology experts is a notable contribution. Such datasets are difficult to obtain in the computational pathology literature because of the high burden for pathologists.

    • Robustness to reduced annotator count. Figure 3 demonstrates that model performance does not degrade dramatically when fewer annotators are available. This is an important practical property to better understand the framework’s applicability in realistic annotation scenarios.

    • Thorough ablation study. The ablation analysis is comprehensive and provides meaningful insight into the contribution of individual components. The authors adequately justify the empirical value of each design choice, which strengthens confidence in the method.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    • Incomplete comparison with SoTA learning from crowds methods. The evaluation does not include recent end-to-end learning from crowds baselines based on transition matrices. Methods such as those learning annotator confusion matrices jointly with the classifier (e. g. , [1],[2],[3]) would provide a more informative comparison. The current set of baselines does not allow the reader to disentangle how much benefit comes from the preprocessing of labels versus the MIL backbone itself. Including these baselines is essential to properly contextualize the contribution and assess whether a fully end-to-end approach would be competitive.

    • Insufficient theoretical motivation for the diagonal structure of the reliability field. In Eq. (1), the weight matrix M_nm is constrained to be diagonal. This effectively reduces the distance modulation to a per-feature rescaling and ignores all inter-feature correlations. The authors do not provide a principled justification for this choice. A diagonal approximation may be computationally convenient, but it requires explicit motivation. Without this, the reliability field as formulated appears to be an ad hoc design decision.

    • The “pushing ambiguous samples further apart” inductive bias is not well-justified. The reliability field Phi is designed to increase distances between samples with high label entropy or feature variance. The implicit assumption is that ambiguous samples should be separated in the clustering step. However, ambiguous samples may naturally occupy a fuzzy intermediate region of the feature space — precisely because the underlying biology is genuinely borderline — and artificially inflating distances between them could fragment meaningful clusters or distort the neighborhood structure.

    • Sparse annotation setting not fully explored. While the experiment with fewer annotators (second strength above) partially addresses annotation efficiency, it still assumes that all available annotators have labeled all samples. In practice, a more realistic and economically motivated scenario is one where annotation effort is distributed across the crowd, so that each sample is only labeled by a subset of annotators. It would substantially strengthen the paper to include an experiment where labels are sparsely assigned — simulating a crowdsourcing protocol — and to evaluate whether the framework remains effective in this setting. This would directly speak to the practical utility of the proposed approach in real annotation pipelines.

    References [1] Li, S. , Xia, X. , Deng, J. , Ge, S. , & Liu, T. (2024). Transferring annotator-and instance-dependent transition matrix for learning from crowds.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11), 7377-7391.
 [2] Chu, Z. , Ma, J. , & Wang, H. (2021, May). Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 7, pp. 5832-5840). 
 [3] Gil-González, J. , Daza-Santacoloma, G. , Cárdenas-Peña, D. , Orozco-Gutiérrez, A. , & Álvarez-Meza, A. (2025). Generalized cross-entropy for learning from crowds based on correlated chained Gaussian processes.  Results in Engineering, 25, 103863.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a clinically relevant and underexplored problem, presents a genuinely multi-annotator WSI dataset, which is difficult to encounter in the literature, and provides a thorough ablation study. However, several technical design choices lack sufficient theoretical justification, the comparison against state-of-the-art learning-from-crowds methods is incomplete (it misses confusion matrices methods as the one I suggested to the authors in my review), and a key practical scenario — sparse, non-exhaustive annotation — is only partially explored, which is of clear importance for posterior deployment.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    The authors have not adequately addressed my concerns.

    • Regarding the lack of baselines, the authors do not clarify why they did not compare their method with the cited literature, nor do they discuss how this absence of comparison affects the validity and positioning of the present study.

    • Regarding the theoretical rationale, the authors reiterate the model workflow instead of providing a clear argument for the conceptual and methodological foundations of the work.

    I believe this work has potential; however, in its current form, I do not consider it suitable for publication at MICCAI. Nevertheless, I encourage the authors to carefully consider the feedback and further strengthen the work for future submissions.



Review #2

  • Please describe the contribution of the paper

    This paper proposes a label reconciliation framework for MIL-based WSI classification with multiple pathologist annotations. It introduces a reliability field that couples bag-level feature variance with label entropy to identify trustworthy neighborhoods, combined with a sample-wise local expert ranking and an adaptive gating mechanism for label fusion. Unlike prior multi-annotator methods that assume global annotator reliability or single-instance data, RaLMPH specifically targets the localized nature of diagnostic authority in clinical pathology.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The motivation is well grounded. Inter-pathologist disagreement on WSI labels is a real bottleneck in clinical pipelines, and existing multi-annotator literature almost exclusively targets single-instance data, leaving the MIL setting unaddressed. The paper’s framing, that pathologist disagreement reflects locally varying expertise rather than global noise, matches how diagnostic workflows actually operate.

    2.The empirical comparison is fairly thorough. RaLMPH is benchmarked against a reasonable set of baselines across two MIL backbones and three datasets, including one real multi-expert clinical cohort.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The method defines an expert’s local reliability based on the cosine similarity between their individual label and the local mean label. However, relying on the mean/consensus to evaluate reliability contradicts the paper’s core motivation that simple consensus is often insufficient or misleading in ambiguous WSI regions. This formulation risks simply upweighting experts who conform to the majority, rather than identifying true specialized diagnostic authority in cases of high disagreement.

    2.According to Sec 3.1, the test set ground truth is determined through “experts’ agreement” from the same cohort. Evaluating a multi-annotator reconciliation method against a consensus generated by the same annotator pool may introduce bias. A more objective evaluation of clinical utility would require an external gold standard, such as IHC results, patient clinical outcomes, or a completely independent panel of senior pathologists.

    3.Table 1 reports the mean and standard deviation for Accuracy and AUC. However, given the relatively small margins of improvement and overlapping standard deviations between RaLMPH and strong baselines, statistical significance tests are necessary to confirm the validity of the performance gains.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses a highly practical problem in computational pathology of handling inter-pathologist variability in MIL, and the proposed reliability-aware framework demonstrates consistent empirical improvements across multiple datasets.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The motivation of this paper is reasonable, and the experimental validation is quite sufficient, so I recommend acceptance.



Review #3

  • Please describe the contribution of the paper

    This paper addresses multi-pathologist label variability in whole-slide image (WSI) classification under Multiple Instance Learning (MIL). The authors propose RaLMPH, a reliability-aware label reconciliation framework that builds a reliability field combining bag-level feature variance and annotator entropy to define a neighborhood metric, ranks locally reliable experts per sample, and fuses opinions via an adaptive gating mechanism. Experiments on a real clinical dataset with six pathologists and on public datasets with synthesized experts show consistent improvements over strong baselines and recent multi-annotator methods.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper introduces a technically novel framework that incorporates a reliability field, local expert sorting, and adaptive gating, enabling more precise modeling of annotator reliability and label fusion at the sample level.

    2.It demonstrates solid experimental rigor, with consistent improvements across multiple MIL backbones, well-designed ablations, and evaluations on both real multi-expert datasets and controlled synthetic settings.

    3.The presentation is clear and well-motivated, with an intuitive pipeline and effective use of figures to illustrate the key components.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The authors cited ‘CONCH’[14] and ‘Madeleine’[9] as two pre-trained feature encoders in the training setting part. But there is no comparison experiments on how they performed differently towards the task.

    2.While the reliability field is a novel concept, its formulation (combining entropy and variance in a Mahalanobis-like metric) may be difficult to interpret clinically or theoretically. How does this concept related to pathological understanding?

    3.While the reported improvements are generally consistent, the experimental evaluation lacks statistical significance testing. The multi-annotator setting primarily relies on synthetic annotators, which raises concerns about real-world applicability. Is it applicable in the clinical scenario?

    4.Is it possible to test on more backbones and compare the performance?

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper tackles an important and underexplored problem in computational pathology, and it is conceptually well-aligned with clinical realities of localized expertise and ambiguous cases. The work would benefit from clearer mathematical/execution details, statistical significance reporting, and broader comparisons to probabilistic multi-annotator fusion adapted to MIL. Nonetheless, the contributions are original and practically relevant, with thorough ablations and useful analyses. I view this as a solid and valuable contribution for MICCAI.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    Although the authors’ response has partially solved my concerns, the proposed method is a novel trial to this area. Therefore, I made my final decision to accept this paper.



Author Feedback

We appreciate the constructive comments. We also appreciate that the reviewers recognized the clinical relevance(R1-3,M) and empirical rigor(R1,R3,M) of RaLMPH. [R1]

We fully concur with your feedback. As noted in the 3rd Para. of the Intro., we cited the recommended [2] as citation [3], with GLAD[25]. Viewing them as a hidden-state approach, we included GLAD for comparison, which utilizes an EM-algorithm for end-to-end training. RaLMPH consistently outperformed GLAD across experiments. We will certainly consider your valuable recommendations in our future work. This design was to enable parallelized feature rescaling while modulating channel-wise operation. Label entropy H acts as a slide-level scalar factor across all channels of z_n, while sigma is computed per channel via feature D.O.[18]. Since [18] is applied to dropped z_n, using WSI encoder[9], the inherent inter-channel correlations captured by the [9] are relevant rather than ignored. We will clarify it in the revised manuscript to address your valuable feedback. We would like to emphasize that the annotations were provided by certified pathologists, whose H captures biological ambiguity. Fig.2 demonstrates that RaLMPH achieves meaningful alignment by integrating both ambiguous WSI features and experts H into the construction of Phi. Our quantitative analysis also confirms this synergy enhanced WSI comprehension. RaLMPH demonstrated robustness in the Sec. 3.3, even when J'<<J. Our study targets clinical expert cohorts where a fixed group of specialists collaborates, which differs from a crowdsourcing where sparse labels from unknown workers are common. We appreciate and concur with your comment. Accordingly, we will reflect this to limitation section and plan to investigate it the furture work. [R2] We appreciate your concern and would like to clarify the following. The reliability field Phi, which precedes the Local Reliable Expert Sorting, incorporates the multiple experts opinions via label entropy H. Consequently, prior to identifying a specific diagnostic authority, Phi determines cluster assignments by comprehensively integrating the insights of all experts. Involving a consistent cohort across the entire splits reflects actual clinical practice, where the goal of clinical MIL is to emulate expert judgment[22]. We also agree with your concern and we proactively evaluated RaLMPH on two public dataset. The consistent performance gains across these benchmarks demonstrate that RaLMPH’s utility is generalizable. We appreciate your suggestion to strengthen our study. We have confirmed a significant improvement p<.05(.01) on the BCNB, While the margins in some metrics of In-house and BRACS may appear small, the consistency of improvement across all 3 datasets and 2 diverse backbones underscores the superiority of RaLMPH. We will add these statistical results to Tab.1 and describe them faithfully in the manuscript. [R3] We apologize for the confusing description. CONCH[14] is a patch-level extractor, while Madeleine[9] is a WSI-level extractor trained on CONCH features. The citation error in Sec. 2.1 (citing [14] instead of [9]) caused the misunderstanding in the experimental description. We have corrected this in the manuscript. Pathologists' opinions on a WSI can vary significantly[24], with levels of disagreement differing across cases[6]. Motivated by this, we developed Phi to jointly consider expert opnions and sample representations. Please see responses to R1-Q2 and Q3 for additional details. Please refer to response to R2-Q3 regarding the statistical test. The In-house dataset (1,382 WSIs, annotated by 6 experts) serves as a real-world validation of RaLMPH, proving its efficacy in meeting actual clinical requirements beyond benchmarks. In this study, we focused on widely-validated MIL frameworks to ensure a reliable comparison. We agree with your suggestion and will actively consider it for follow-up studies.



Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper received mixed reviews, including two weak accepts and one weak reject. The work addresses a clinically important and relatively underexplored problem: handling multi-pathologist disagreement in MIL-based WSI classification. The positive reviews recognize the strong motivation, the value of using a real multi-expert clinical cohort, and the promise of the proposed reliability-aware label reconciliation framework, with empirical improvements and ablation studies providing some support for the main design choices. However, the reviewers also raise important concerns regarding incomplete comparison with recent learning-from-crowds methods, insufficient theoretical or clinical justification for the reliability-field formulation, and potential bias in the evaluation protocol due to the use of consensus labels from the same annotator pool as the test reference. The paper would also benefit from statistical significance testing and further analysis under realistic sparse annotation settings where not every annotator labels every sample. Overall, the paper has meaningful clinical motivation, a valuable problem setting, and encouraging results, but several aspects of the formulation and evaluation remain insufficiently justified; therefore, the authors are invited to submit a rebuttal before a final decision is made.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After rebuttal, this paper receives two accept recommendations and one rejection. The AC acknowledges that RaLMPH addresses an important and underexplored problem in computational pathology: reconciling multi-pathologist disagreement for MIL-based WSI classification. The paper is clinically well motivated, uses a valuable real-world cohort annotated by six pathologists, and proposes a reliability-aware framework that models local expert reliability rather than relying on simple consensus. The rebuttal addresses several key concerns by clarifying the comparison with learning-from-crowds methods, the rationale for the reliability field, the evaluation protocol, statistical testing, and the scope of sparse-annotation settings. Although some concerns remain regarding theoretical justification and broader comparison with recent annotator-confusion models, the majority of reviewers agree that the method is novel, practically relevant, and empirically validated across multiple datasets and MIL backbones. Therefore, the AC recommends acceptance.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Overall, the reviewers acknowledge the strong clinical motivation, novel formulation, and generally solid empirical performance, with two reviewers supporting acceptance after rebuttal. While some concerns remain regarding incomplete comparisons, limited theoretical justification, and potential evaluation bias, the rebuttal partially addressed these issues and strengthened confidence in the methodology.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper proposes RaLMPH, a reliability-aware multiple-instance learning framework for WSI classification under multi-pathologist disagreement. The method models local feature-space neighborhoods together with annotator-specific reliability estimates to derive reliable pseudo-labels or harmonized labels for ambiguous samples, and evaluates the approach on a real WSI dataset annotated by six pathology experts as well as public datasets with synthetic multi-rater labels.

    The initial reviews were mixed but overall close to positive, with scores of 3 / 4 / 4.Reviewers appreciated the clinical relevance of the problem, the use of a real multi-pathologist WSI dataset, the attempt to model localized diagnostic reliability rather than global annotator reliability, the empirical improvements across MIL backbones/datasets, and the ablation analyses. The main concerns were incomplete comparison to recent learning-from-crowds baselines, insufficient theoretical justification for the reliability-field formulation and its diagonal/metric structure, possible bias in using expert agreement from the same cohort as ground truth, lack of sparse-annotation evaluation, limited statistical significance reporting, and incomplete comparison of pretrained feature encoders.

    The rebuttal addressed several of these concerns. The authors clarified that GLAD was included as a comparison, explained the design rationale for the reliability field and channel-wise feature rescaling, emphasized that expert entropy reflects biological ambiguity, clarified that the method targets a fixed expert cohort rather than generic crowdsourcing, and provided additional explanation regarding statistical testing and feature extractor usage. They also noted that the in-house dataset contains 1,382 WSIs annotated by six experts and argued that results on public datasets support generalization.

    Some concerns remain. Reviewer 1 was not persuaded that the rebuttal adequately addressed the missing learning-from-crowds comparisons or the conceptual justification of the reliability-field formulation. The evaluation protocol still relies partly on expert agreement as the target and does not fully resolve the concern about an independent gold standard. The sparse-annotation setting is also left largely as future work. Thus, the paper remains somewhat borderline.

    After rebuttal, the reviewer opinions are R / A / A. Reviewer 1 maintained reject due to unresolved concerns about baselines and theoretical rationale. Reviewers 2 and 3 recommend accept, emphasizing that the problem is clinically relevant, the motivation is reasonable, the experimental validation is generally sufficient, and the method is a novel attempt in an important area. I place slightly greater weight on the two positive post-rebuttal assessments and the practical relevance of the real multi-pathologist WSI setting, while acknowledging the unresolved conceptual and evaluation limitations.

    I recommend Accept. The paper addresses an important and clinically realistic problem, and the use of real multi-expert pathology annotations is valuable. Although the theoretical justification and comparison to recent learning-from-crowds approaches should be strengthened, the reviewer consensus after rebuttal is moderately positive, and the remaining issues appear to be limitations rather than fatal flaws. The camera-ready version should clearly discuss the assumptions behind the reliability field, the limitations of using expert agreement as reference, the lack of sparse-label evaluation, and the scope of comparison to learning-from-crowds methods.



back to top