List of Papers Browse by Subject Areas Author List
Abstract
Accurate nucleus detection in pathology images is crucial for disease diagnosis. Deep learning based methods require extensive annotations of nuclei, which are time-consuming for pathologists. Active learning (AL) provides an attractive paradigm for reducing annotation efforts by iteratively selecting the most valuable samples for annotation. However, most AL methods do not consider utilizing crowdsourced annotations from multiple workers with varying expertise levels and labeling costs, limiting their practical applicability. Recent approaches design AL strategies that adaptively select the most cost-effective worker for each sample, but these methods solely focus on the classification task, overlooking the development of an AL framework for the detection task. Additionally, they struggle to adapt to the changes in model performance during AL iterations, resulting in inefficiencies in sample selection and cost management. Based on the above considerations, we propose C2AL, a novel cost-effective AL framework
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2025/paper/3012_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{TanJia_Costeffective_MICCAI2025,
author = { Tang, Jiao and Zu, Yuankun and Zhu, Qi and Wan, Peng and Zhang, Daoqiang and Shao, Wei},
title = { { Cost-effective Active Learning for Nucleus Detection Using Crowdsourced Annotations with Dynamic Weighting Adjustment } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2025},
year = {2025},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15972},
month = {September},
page = {95 -- 105}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a cost-effective active learning (AL) framework for nucleus detection using crowdsourced annotation. The framework integrates sample uncertainty, annotator credibility, and cost.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) One novel point is the method of selecting image-worker pairs jointly based on three criteria: sample uncertainty, annotator credibility, and cost in a single framework. 2) The use of a score function with dynamic weights that adapt over AL cycles based on model confidence.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1) Overall technical contribution of this study is limited. Dynamic adjustment strategies like curriculum or uncertainty-aware AL are not new in principle. I will suggest to the authors to please emphasize what specifically is novel beyond combining known ideas. 2) Related work section doesn’t cover the recently published studies. 3) Results are not properly presented and causing difficulty to compredend, i.e., ablation study can be more simplified for better readability. 4) Study is compared with old techniques whereas it should have been compared with recently published ones, i.e., [R1] 5) Authors didn’t justify well the contribution made of each component in the ablation study. i.e., Authors should try to compare with other some technique to identify informative samples.
[R1] Tang, J., Yue, Y., Wan, P., Wang, M., Zhang, D., & Shao, W. (2024, October). OSAL-ND: Open-Set Active Learning for Nucleus Detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 351-361). Cham: Springer Nature Switzerland.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2025/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(1) Strong Reject — must be rejected due to major flaws
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors use Faster R-CNN as the detection model throughout the experiments. While this model remains a strong baseline, it is significantly outpaced in recent years by more modern detectors such as YOLOv8, DINO, or RT-DETR, which offer improved speed and accuracy. It would strengthen the paper if the authors justify the choice of Faster R-CNN (e.g., due to consistency with prior work or better interpretability in medical settings). Secondly, the comparative analysis is weak. The proposed method is only compared against older baselines, and lacks evaluation against more recently published state-of-the-art approaches. Additionally, the contribution is not sufficiently justified in light of these limited comparisons. These issues collectively lead me to the decision to reject the paper.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
After reviewing the author rebuttal, I acknowledge that the paper presents a more clearly articulated and practically valuable contribution than initially perceived. The integration of multi-criteria sample-worker selection and dynamic weighting adjustment tailored for crowdsourced nucleus detection represents a thoughtful extension of existing active learning methods. The experimental results support the framework’s effectiveness in improving cost-efficiency and accuracy. I now believe the work makes a meaningful contribution to practical AL applications in medical imaging, and I revise my recommendation accordingly.
Review #2
- Please describe the contribution of the paper
The paper proposes C2AL, a novel cost-effective Active Learning (AL) framework tailored for nucleus detection tasks in pathology images under a crowdsourced annotation environment. The key innovations include a dynamic weighting strategy to balance sample uncertainty, worker credibility, and annotation costs effectively throughout AL iterations.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The introduction of dynamic weighting adjustment for sample-worker pair selection effectively addresses variations in model confidence across AL iterations. Extends active learning from typical classification problems to nucleus detection, incorporating both classification and localization, which is novel and significantly enhances applicability in medical imaging. Comprehensive evaluations on real and simulated datasets demonstrate the framework’s effectiveness in improving detection accuracy and reducing annotation costs compared to current methods. Explicitly models worker credibility and annotation costs, making the framework practically valuable in realistic crowdsourced scenarios.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The SimNuCLS and SimLizard datasets are artificial. Although useful, they may not entirely capture real-world complexities, potentially limiting generalization. While the method is novel, the complexity and practicality of implementing the dynamic weighting adjustment in large-scale or real-world scenarios could pose challenges.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2025/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — could be accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The innovative dynamic weighting adjustment and its successful application to nucleus detection significantly advance the field. Robust experiments and clearly demonstrated cost-effectiveness underscore the practical value of this work.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The Authors propose a novel Active Learning (AL) framework, called C2AL, for detecting nuclei in a crowdsourced environment. The framework is tested on NuCLS, SimNuCLS, and SimLizard, and shows improved performance compared to baseline methods, as indicated by the mAP curves. Additionally, an ablation study is conducted to highlight the importance of selected factors, and a comparison using visual results is provided, which helps better appreciate the outcomes of C2AL.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel Framework: The authors propose an AL framework that incorporates criteria such as sample uncertainty, worker credibility, worker cost, and a score function with dynamic weighting adjustment.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
-
Claim of Novelty – Existing Work: The claim that this is the “first” AL framework for nucleus detection in a crowdsourced environment might be too strong, as there are existing methods, such as NuCLS, that address similar challenges. The Authors should acknowledge these prior works, including those that inspired or contributed to methods like NuCLS. If their framework introduces significant innovations or improvements over existing approaches, these differences should be clearly highlighted to justify the novelty of the current work.
-
Unprecise Claim – AL Framework: The statement in the Introduction that “the development of an AL framework for the detection task (i.e., nucleus detection) is overlooked” seems too strong. For instance, there is already a work presented at MICCAI 2024 on this topic, even if not specifically focused on the crowdsourced environment: Tang, Jiao, et al. “OSAL-ND: Open-Set Active Learning for Nucleus Detection,” MICCAI 2024. Thus, I suggest that the Authors explore the literature more thoroughly for this sub-task and, if possible, include key references on the topic.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2025/en/REVIEWER-GUIDELINES.html
The following comments are divided by section, figure, or table for clarity:
- Introduction
-
References for DWA: The “dynamic weighting adjustment” (DWA) is used for adjusting the importance of each sample-worker. If this is used or inspired by others scenarios, appropriate references should be provided.
-
Terminology “Entropy”: Clarify whether the method uses “entropy” in general or “maximum entropy sampling” specifically, as referenced in [21].
Figure 1
- Image source – Attribution: Some visual components in Figure 1 (e.g., the H&E stained cells) appear similar to Figure 1 in Tang, Jiao, et al. OSAL-ND: Open-Set Active Learning for Nucleus Detection, MICCAI 2024. If elements are reused or inspired by this work, this should be explicitly acknowledged in the figure caption or manuscript.
2.2 Worker Reliability
- Similarity metric – Selection criteria: The method “we select the K most similar images,” but the metric used to compute similarity is not clear defined. Authors should specify the similarity metric (e.g., cosine similarity, …).
2.4 Score Function
- Sigmoid range – Readability: When defining weights with sigmoid functions, it is recommended to clearly state the intended range (e.g., [0, 1]) to improve clarity for the reader.
- Experiment Typo – Grammar: Correct “dose no” to “does not.”
Bounding box shift – Clarification: The reason for splitting the shift amount S_{a_i} by B_{o_i} (i.e., positions of a certain percentage of bounding boxes) needs further explanation to justify the design choice.
-
Model citation – Faster R-CNN: When referencing the use of Faster R-CNN, it would strengthen the paper to cite foundational and/or related works that apply this model to similar tasks.
-
Typo – Spelling: Replace “mutli” with “multi.”
-
Dataset details: Double-check the number of FOVs: NuCLS reportedly has 52 (not 53), and Lizard has 291 (not 208), according to the original dataset sources.
-
Style – Formal tone: Replace informal terms such as “obviously” with more formal alternatives suitable for scientific writing.
For future work, I would recommend: 2.3 Worker Costs
- Cost weighting – Nucleus types: The definition of C_i could be improved by incorporating a balancing factor for different nucleus types, as they may inherently carry different annotation costs or complexities.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — could be accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I thank the Authors for proposing C2AL to address the problem of detecting nuclei in a crowdsourced environment. While I appreciate the effort in comparing C2AL with other baseline techniques, I suggest exercising caution with the claim of it being the ‘first’ AL framework for this scenario.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I thank the Authors for addressing the questions. The main concerns I had have been resolved, and I am pleased with the Authors’ plans to implement the proposed adjustments. Therefore, I recommend acceptance.
Author Feedback
We thank all reviewers for their comments, and appreciate the overall enthusiasm on our work. For R#1 “Artificial datasets” ==Currently, publicly available crowdsourced datasets for nucleus detection are extremely limited. Aside from NuCLS, there are no other widely accessible real-world crowdsourced datasets, which restricts the ability to capture real-world annotation variability. As part of our future work, we plan to construct comprehensive real-world crowdsourced datasets to further advance this field. “Complexity and practicality” ==We acknowledge that the dynamic weighting adjustment introduces additional complexity to the active learning framework. However, this approach is designed to improve cost efficiency according to nuclei detection performance of AL at each cycle, which is critical in real-world crowdsourced scenarios. For R#2 “Novelty” ==We introduce a multi-criteria sample-worker selection process that considers sample uncertainty, worker cost, and worker credibility in each AL cycle. Furthermore, we introduce dynamic weighting adjustment, a strategy that adjusts the importance of each sample-worker pair based on the model’s progress during the learning process. Unlike conventional uncertainty-aware approaches, our method incorporates multiple criteria beyond just sample uncertainty, ensuring more balanced and cost-effective sample selection. “Another Comparison” ==[R1] introduced an AL framework under the open-set environment, while our approach is specifically designed for AL in crowdsourced environments, where varying worker expertise, labeling costs, and credibility must be considered. [R1] Tang, J., Yue, Y., Wan, P., Wang, M., Zhang, D., & Shao, W. (2024, October). OSAL-ND: Open-Set Active Learning for Nucleus Detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 351-361). Cham: Springer Nature Switzerland. “Ablation Study” ==To further evaluate the effectiveness of C2AL, we compare it with several variants: w/o F, w/o C, w/o R, w/o m&o, w/o DWA. The results demonstrate the importance of each component in achieving optimal performance. For R#3 “Imprecise claim” ==We acknowledge that NuCLS is a significant crowdsourcing effort for nucleus detection, but it does not specifically introduce an active learning (AL) framework tailored to real-world crowdsourced environments for reducing annotation costs. Therefore, we maintain our claim that this is the first AL framework for nucleus detection specifically designed for crowdsourced settings. Additionally, we recognize that the statement in the Introduction regarding the lack of AL frameworks for nucleus detection tasks was too broad. Our focus is on AL frameworks within crowdsourced environments, which we will clarify in the revised manuscript, along with a more comprehensive review of related sub-tasks. Thank you for highlighting this point. “Clarification of details” ==Thank you for carefully pointing out the issues in the text and figures. We will revise the manuscript to address these details accordingly.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- If your recommendation is “Provisional Reject”, then summarize the factors that went into this decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. You do not need to provide a justification for a recommendation of “Provisional Accept” or “Invite for Rebuttal”.
N/A
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
This manuscript introduces a cost-effective active learning framework for nucleus detection using crowdsourced annotations. It adopts a dynamic weighting adjusted score function, which combines sample uncertainty, worker credibility and worker cost, to select the most cost-effective sample-worker pairs for data annotation. The framework produces better experimental results than several existing relevant approaches on both simulated and real crowdsourced datasets. The rebuttal has addressed the reviewers’ concerns regarding the method’s technical novelty, complexity/practicality, usage of artificial datasets, ablation study, and statement of the contributions. Thus, the manuscript is recommended for acceptance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A