Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Deep neural networks demand large-scale labeled dataset for optimal performance, yet the cost of annotation remains high. Deep active learning (DAL) offers a promising approach to reduce annotation cost while maintaining performance. However, traditional DAL methods often fail to balance performance and computational efficiency, and overlook the challenge posed by class imbalance. To address these challenges, we propose a novel framework, named Class- Balancing Deep Active Learning(CB-DAL), comprising two key modules: auto-mode feature mixing(Auto-FM) and minority push-pull sampling(MPPS). Auto-FM identifies informative samples by simply detecting in inconsistencies in predicted labels after feature mixing, while MPPS mitigates the class imbalance within the selected training pool by selecting candidates whose features close to the minority class centroid while distant from features of the labelled majority class. Evaluated across varying class imbalance ratios and dataset scales, CB-DAL outperforms traditional DAL methods and the counterparts designed for imbalanced dataset. Our method provides a simple yet effective solution to the class imbalance problem in DAL ,with broad potential applications.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2541_paper.pdf

SharedIt Link: https://rdcu.be/dY6fv

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72390-2_2

Supplementary Material: N/A

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Lin_ClassBalancing_MICCAI2024,
        author = { Lin, Hongxin and Zhang, Chu and Wang, Mingyu and Huang, Bin and Shao, Jingjing and Zhang, Jinxiang and Gao, Zhenhua and Diao, Xianfen and Huang, Bingsheng},
        title = { { Class-Balancing Deep Active Learning with Auto-Feature Mixing and Minority Push-Pull Sampling } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15012},
        month = {October},
        page = {14 -- 22}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes a novel framework called Class-Balancing Deep Active Learning (CB-DAL) to address the challenges of high annotation costs, computational efficiency, and class imbalance in deep active learning.

    CB-DAL consists of two key modules: auto-mode feature mixing (Auto-FM) and minority push-pull sampling (MPPS). Auto-FM identifies informative samples by detecting inconsistencies in predicted labels after feature mixing, while MPPS mitigates class imbalance by selecting candidates whose features are close to the minority class centroid and distant from the labeled majority class features.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. A novel approach to Class-Balancing Deep Active Learning has been proposed

    2.The innovation points of this text are valuable for solving the problem of imbalanced medical data categories and the difficulty in obtaining labeled medical data.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1.The first of the innovations in this paper, Auto-Feature Mixing, is quite similar to the ALFA-Mix method in this CVPR 2022 paper and the formulas have barely changed. so the innovativeness of this part is relatively weak.

    Parvaneh, A., Abbasnejad, E., Teney, D., Haffari, G. (Reza), van den Hengel, A., Shi, J.Q.: Active Learning by Feature Mixing. In: Presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12237–12246. (2022)

    1. The second innovation, Minority Push-Pull Sampling, may not be able to effectively improve the discrimination ability between minority classes when the dataset exhibits a long-tail distribution. Additionally, all the experiments in this paper are binary classification problems, and there is no use of more classes to demonstrate the effectiveness of this module.

    3.The experiments in this paper are not very comprehensive. Most of the experiments were conducted on relatively small private datasets ( external test set :osteolytic OS:12, GCT:42)). Furthermore, On the more imbalanced public dataset ISIC2020, this paper only tests one state-of-the-art method, DAL, and do not test more state-of-the-art methods.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The composition of this article should be more systematically organized. Please provide a more detailed explanation of the preliminary between the ratio of unlabeled samples and Class-Balancing Deep Active Learning. It may be difficult to fully grasp the relationship between them the first time reading this paper.

    This paper can summarize relevant work, especially very similar work, highlighting the innovation of this paper.

    Additionally, enhance the description of the experiments conducted on ISIC2020 dataset. The paper can attempt to evaluate the model’s performance on datasets with more diverse classes, rather than solely relying on binary classification datasets.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The innovations presented in this paper are not substantial enough, and the experiments conducted are not sufficiently comprehensive.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • [Post rebuttal] Please justify your decision

    The author’s response partially resolved my confusion, but the description of the ALFA-Mix module innovations in the original manuscript was not clear, I decided to maintain my opinion



Review #2

  • Please describe the contribution of the paper

    To mitigate the cost of annotating a large-scale dataset of medical images, deep active learning offers a promising approach to select informative samples from the dataset for annotation. However, existing approaches often suffer from heavy computational costs and also overlook the class imbalance problem. This work provides a novel framework for deep active learning by incorporating two aspects. One is auto-mode feature mixing to extract informative samples by mixing features of labeled and unlabeled samples. The other is minority push-pull sampling to effectively select minority samples. Based on comparative studies, the superiority of the proposed method was confirmed, particularly in situations where the ratio of annotated samples is low in the dataset.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    • The novel approach to sampling informative instances from the dataset by utilizing the mixed feature representation of labeled and unlabeled instances, along with minority information, seems reasonable.

    • Quantitatively superior results based on the comparison with existing methods strengthen the significance of this work.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • A more detailed description of the existing approaches, such as BAL and ALFA-Mix, will enhance readers’ comprehension.

    • Fig. 1 and Fig. 2 contain a lot of visual information; however, the brief captions make it difficult to understand the contents.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    More detailed information on the figures and existing approaches would enhance the presentation of this work.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The reasonable approach to deep active learning and compelling quantitative results support the significance of this work.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The objective of this research paper is to present a novel method for active learning in the domain of medical image analysis. The primary contributions are twofold. Firstly, it introduces a module named Auto-FM, which identifies informative unlabeled samples for manual annotation, employing a technique with low computational resource requirements. Secondly, by leveraging a module termed MPPS, this methodology endeavors to mitigate the challenge posed by class imbalance, a common issue in numerous real-world datasets.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1- The proposed method is motivated and the methodology is sound and intuitive. 2- Both the Auto-FM and MPPS modules work together to add informative samples while also maintaining the balance between classes. 3- The experiments and ablations are well designed to demonstrate the effectiveness of each module in the method.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1- In equation (5), $c_t^{minor}$ and $c_t^{maj}$ are not defined. More specifically, what is the $t$ subscript? 2- The Auto-FM section lacks clarity in regards to how the feature mixing process is optimized. 3- In all of the experiments the datasets contain only data with binary labels, it would be interesting to see if the class-balancing technique would work well when there are more than 2 classes.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    1- I would suggest that the authors provide more clarity on the Auto-FM and how the feature mixing process is optimized. 2- If it is possible to experiment on datasets with more than 2 classes, the impact of MPPS module would be more clear.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The research paper presents a novel approach to enhance active learning techniques for analyzing medical images. This work is valuable due to the scarcity of annotated data in the medical field. The proposed method incorporates two components that aim to augment the labeled dataset with more informative samples while maintaining a balance across different classes. The methodology is straightforward, and the results demonstrate substantial performance gains over prior techniques. Given the practical nature of the problem and the simplicity and effectiveness of the solution, I would recommend accepting the paper for publication.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    After seeing all reviews and also authors rebuttal I update the rate.



Author Feedback

Thanks to all the reviewers for your valuable comments on our work. Especially, thanks to R3 and R4 for accepting the paper directly. We appreciate your recognition of:

  1. Novelty (R1: “a novel framework” R3: “a novel method” R4: “the novel approach”)
  2. Well-designed experiments, superior results (R3: “the experiments and ablations are well designed”; “substantial performance gains over prior techniques” R4: “Quantitatively superior results”;)
  3. Well-organization (R3: “Very Good” R4: “Good”)

Q1: Relatively weak innovativeness of Auto-FM (R1). A1: We improved ALFA-Mix[15] by making the mixing matrix α a learnable parameter, thereby simplifying the solving process and reducing computational demands. This is the first time that the strengths of learnable feature interpolation have been leveraged to identify informative samples in medical image analysis involving high-resolution images. Q2: Suggestion about evaluation on imbalanced multi-class datasets (R1&R3). A2: Extending to multi-class tasks is valuable, but it does not undermine our approach’s proven effectiveness for investigated imbalanced binary problems, which are common in medical research. Additionally, the MPPS module can be readily extended by assigning pseudo-labels based on the predicted probabilities of multi-class data, sorting accordingly, and sampling according to Eq.5. This iterative approach updates class distributions, emphasizing tail classes while maintaining balance. We appreciate the reviewers’ suggestions and will evaluate multi-class performance in future work. Q3: Not very comprehensive experiments (R1). A3: For comprehensive experiments, we validated our method across various data scales and class imbalance ratios. Specifically, we tested it on 4 subsets of our private dataset with different imbalance rates and the large-scale public dataset ISIC2020. (Sec3.2)

  • Small but real-world private datasets. While our private datasets are relatively small, they effectively demonstrate our method’s real-world performance for low-incidence diseases where large-scale data is difficult to obtain.
  • Sufficient comparisons. We conducted a fair comparison with BAL [10] (the latest SOTA) under the same setting on ISIC2020, equally demonstrating superiority over other methods. Q4: Explanation of the preliminary between the ratio of unlabeled samples and CB-DAL (R1). A4: We explored different ratios of unlabeled samples to investigate their impact on model performance and validate our method’s effectiveness with less data. The sampling rules and settings for different ratio of samples have been described in Sec3.2. Q5: Clarifications about Eq.5 (R3). A5: $c_t^{minor}$ denotes features of majority classes, $c_t^{maj}$ denotes features of minority classes, and $t$ indicates the current $t$-th round of iteration. Q6: Clarifications of Auto-FM optimization(R3). A6: We optimize the feature mixing process using Eq.4, comprising two parts: 1) 𝑙(𝑓(𝑍𝑎,𝑦∗)) (Eq.3), with its complement used as the loss to maximize the feature discrepancy between unlabeled and labeled samples while avoiding excessive sensitivity to the discrepancy; 2) The addition of the L1 norm of 𝛼 to prevent overfitting in the feature mixing process. Q7: More detailed description of ALFA-Mix and BAL (R4). A7: ALFA-Mix[15] proposed feature mixing to discover valuable samples with lower computational complexity than the SOTA BADGE, which is crucial for high-resolution medical images. BAL[10] employs a pretrained VAE to model feature space distribution and introduces Copulas for evaluating minority class probabilities to enable balanced sampling. Q8: More detailed information on the Fig.1&Fig.2 for better presentation (R4). A8: Our intention is to efficiently convey the work through these two figures. Fig.1 highlights existing DAL limitations and our improvements; Fig. 2 illustrates details about feature mixing in Auto-FM, minority-focused sample selection and the iterative training process.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The authors have addressed the reviewers’ most pressing concerns in their rebuttal. However, the novelty of modules in the original manuscript should be clearly articulated.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The authors have addressed the reviewers’ most pressing concerns in their rebuttal. However, the novelty of modules in the original manuscript should be clearly articulated.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The authors have responded well to the reviewers’ concerns. If accepted, I would encourage the authors to revise the camera-ready version in accordance with the reviewers’ recommendations to enhance clarity.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The authors have responded well to the reviewers’ concerns. If accepted, I would encourage the authors to revise the camera-ready version in accordance with the reviewers’ recommendations to enhance clarity.



back to top