List of Papers Browse by Subject Areas Author List
Abstract
The scarcity of data in medical image classification using deep learning often leads to overfitting the training data. Research indicates that self-distillation techniques, particularly those employing mean teacher ensembling, can alleviate this issue. However, directly transferring knowledge distillation (KD) from computer vision to medical image classification yields subpar results due to higher intra-class variance and class imbalance in medical images. This can cause supervised and contrastive learning-based solutions to become biased towards the majority class, resulting in misclassification. To address this, we propose UDCD, an uncertainty-driven contrastive learning-based self-distillation framework that regulates the transfer of contrastive and supervised knowledge, ensuring only relevant knowledge is transferred from the teacher to the student for fine-grained knowledge transfer. By controlling the outcome of the transferable contrastive and teacher’s supervised knowledge based on confidence levels, our framework better classifies images under higher intra- and inter-relation constraints with class imbalance raised due to data scarcity, distilling only useful knowledge to the student. Extensive experiments conducted on benchmark datasets such as HAM10000 and APTOS validate the superiority of our proposed method. The code is available at https://github.com/philsaurabh/UDCD_MICCAI.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1765_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1765_supp.pdf
Link to the Code Repository
https://github.com/philsaurabh/UDCD_MICCAI
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Sha_Confidence_MICCAI2024,
author = { Sharma, Saurabh and Kumar, Atul and Chandra, Joydeep},
title = { { Confidence Matters: Enhancing Medical Image Classification Through Uncertainty-Driven Contrastive Self-Distillation } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15010},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper focuses on medical image classification using contrastive learning and EMA-based distillation. Its primary contribution lies in integrating confidence/uncertainty into the EMA-based contrastive learning framework for medical image classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well written and well organized.
- This paper demonstrates good performance compared to the existing methods.
- The illustration of the framework is intuitive and easy to understand.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
This method has been widely explored before, since the concepts of contrastive learning, relation matrix, and EMA-based distillation have been extensive validated before. Take [1], a method two years ago, as an example, it proposed uncertainty-aware contrastive distillation method with a relation matrix between representations of pixels for improving visual recognition. Thereby, the technical contribution of this paper is not enough.
-
The figure 3 is strange, as it uses different colors for three subfigures but uses the sample color in figure 3(b) for different ablation studies. Also, the legend is too small.
-
How does the proposed method solve the problem in the motivation is not clear. What’s the definition of relevant knowledge and useful knowledge? Moreover, what’s the relationship between the proposed method and the problem the class-imblance? The ambiguity of these issues leads to insufficient motivation in this article.
-
It appears that the supplementary material spans 4 pages, doubling the specified limit (refer to the guideline). This could potentially be unfair to other paper submitters.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Please see the paper weakness.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Reject — should be rejected, independent of rebuttal (2)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Reject recommendation due to insufficient technical contribution, ambiguous motivation, and violations of page count guidelines.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper presents a novel framework called Uncertainty-Driven Contrastive Self-Distillation (UDCD), which addresses the challenges of high intra-class variance and class imbalance in medical image classification. The UDCD framework enhances the transfer of knowledge by using an uncertainty metric to regulate the knowledge distillation process from a teacher model to a student model based on confidence levels. This approach allows for fine-grained knowledge transfer, particularly useful in medical image datasets where data scarcity and class imbalance are prevalent. The framework’s efficacy is validated through extensive experiments on benchmark datasets like HAM10000 and APTOS, where it demonstrated significant improvements over existing methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The methodological design innovatively addresses the issue of class imbalance inherent in medical data, employing targeted strategies to enhance performance.
-
The experimental section includes extensive comparative analyses across multiple datasets, effectively demonstrating the efficacy of the approach.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The method design is complex, with the overall optimization objective in Equation (8) involving numerous hyperparameters. However, the experimental section lacks an analysis of the sensitivity to these hyperparameters, suggesting that the method might require intricate parameter tuning, potentially affecting its applicability.
-
The method section does not sufficiently highlight the key aspects, with much of the content covering techniques commonly used in knowledge distillation or transfer learning. The Contrastive Relation Matrix (CRM) component, which is a novel element, should be emphasized more, including its design motivation and implementation details.
-
In the experimental section, the legends within the figures are too small, and the color contrast is not sufficiently pronounced. Additionally, the results presented in the supplementary material are not referenced in the main text, which could lead to an incomplete understanding of the findings.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
This paper does not give the key hyperparameter choices for the method and reproducible code in either the main text or the supplementary material, and the current version is not well reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Please see the weaknesses part.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The issues addressed in this paper are significant, and it proposes a viable solution; however, there are concerns regarding the contribution of the method design and the experimental validation of its effectiveness. The current version of the manuscript is not yet sufficient for acceptance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes to use uncertainty to balance the influence of transferable contrastive knowledge and the supervised knowledge of the teacher during self-distillation style machine learning. It calls this approach Uncertainty-Driven Contrastive Self-Distillation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper presents a method for contrastive learning that takes uncertainty into account
- Results reported clearly show that the proposed method outperforms other approaches in a set of evaluation tasks.
- Instead of the distribution the approach represents each class by a single anchor to avoid bias caused by class imbalance. This makes sense, but I wonder if this is not oversimplifying the distributions in the feature space. Is there no benefit in using more samples to represent this distribution? Maybe I am misunderstanding this part.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper was not easy to follow. Specifically, the method section should be improved (see point 10 below). Currently it is not easy to follow, or make out the exact difference to existing approaches. I understand that the balancing based on uncertainty is the key, but it is hard to follow.
- Can you indicate which sub-section / part of the method section refers to which part of the figure? having more clearly visually separated blocks - if this is possible - would be helpful for following the approach explanation.
- It is not easy to follow what is happening in the feature representation and what is relevant downstream in the part where classes play a role. From the perspective of a reader who is not deeply involved in this area, but might still benefit from the papers result, this should be improved.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- Please separate rationale and motivation from the description of the method, and try to provide a path from training inputs to model, and from input to inference output in the method description, this makes reading easier. The rationale and limitation of existing models might be better described in the introduction or the conclusion.
- What is „dark knowledge“? Please make sure that terms in the paper are explained so that also readers who are not very close to the particular area can follow.
- Please explain how the proposed model and the alternatives for the evaluation were implemented. Was there really only the balancing that was different, or did the models differ in more aspects. That would make it harder to understand the contribution of UDCD. Please clarify this point.
- What is the interpretation of the t-SNE graphs? I assume optimally you would like to have one class per cluster and vice versa, but I am not sure. Please clarify this part.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents an interesting method and a good experimental evaluation, comparing the approach to alternatives. The weak point of the paper is that it is not easy to follow.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We sincerely thank the reviewers for their valuable comments on our paper. Below are our collective responses to the concerns raised: Technical contribution: We employ supervised contrastive features, which have not been utilized in prior works. The derived metrics are based solely on supervised contrastive learning (SCL), representing a novel technical contribution. Additionally, the uncertainty and memory components are designed to mitigate bias towards majority class samples. Our proposed solution integrates SCL with uncertainty-based transfer and is clearly outlined in Section 1.
Emphasis on CRM: While we acknowledge CRM as a novel component contributing to uncertainty, our overall novel contribution encompasses the integration of various components. Due to limited space, we aimed to highlight each component rather than focusing solely on CRM.
Figures & Labels, Complexity of the article: We sincerely apologize if the manuscript was challenging to follow. Additionally, the Figure s 3(a-c) in the manuscript utilize different color schemes because they represent different analyses. Due to the space constraints and the need to present extensive analysis, we had to condense the information, which required us to use more complex terminology and brief descriptions. Also, we combined the rationale and motivation for conciseness, which may have contributed to the complexity of the paper. Furthermore, the figures were adjusted to fit the available size, resulting in smaller labels. Based on the feedback, we will provide figures with enhanced label sizes in the final drafts for publication, if permitted. Regarding the suggestion to subsection parts for better description of Figure 1 and provide separate rationale and limitations, we respect the comment. However, for Figure 1, we have provided a detailed and stepwise elaborative description in section 4, along with an algorithm in the supplementary material and also as per guidelines, we are unable to change the content in the original paper.
Equation 8: We respectfully disagree with the comment regarding Equation 8. The equation explicitly lists three hyperparameters, denoted by “lambda”s, which is considered an optimal number according to the literature. The remaining parameters are confidence parameters that are learned automatically through uncertainty during the learning process, as mentioned just above Equation 7.
Dark, useful and relevant Knowledge: We apologize for not providing a more in-depth description of these terms. “Dark knowledge” is commonly used in contrastive learning to refer to complementary knowledge or secondary probabilities used for incorrect classes. Due to its prevalence in the literature, we did not elaborate on it. The terms “relevant” and “useful” knowledge are used in their literal sense, referring to the knowledge necessary for refining the student model’s learning.
tSNE: The reviewer’s assumption is indeed accurate, also supported by Section 3.2.
Implementation: We adhered to the settings outlined in the literature, as detailed in Section 3.1, and substituted the algorithms with our own. To assess the efficacy in handling class imbalance, we sampled input data with varying distributions and ran all models separately on the same data.
Supplementary and Codes: We apologize for this unintentional error caused by formatting issues that resulted in content being moved to other pages. Since this content is not part of the main manuscript, we assure you that, in accordance with the guidelines, we will ensure it is correctly positioned within the specified range for the camera-ready version. Additionally, we respectfully disagree with the comment regarding an incomplete understanding of the supplementary results. The results presented in the paper suffice for our analysis. We have provided detailed headings in the supplementary material that reference the corresponding paragraphs in the paper for better understanding & algorithm for reproducibility.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The paper introduces Uncertainty-Driven Contrastive Self-Distillation (UDCD), which integrates confidence/uncertainty into the EMA-based contrastive learning framework for medical image classification, offering a novel approach to balance the influence of transferable contrastive knowledge and supervised knowledge during self-distillation. However, similar as Reviewer4 and Reviewer8, I have some major concerns regarding the contribution of the method design and the experimental validation of its effectiveness. The claim that SCL (Spatio-temporal Contrastive Learning) is novel for this task is inaccurate, given that the CRCKD dataset also utilized SCL. Moreover, the paper lacks clarity, making it difficult to follow, and the supplementary file exceeds the page limit. Consequently, I recommend rejection.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The paper introduces Uncertainty-Driven Contrastive Self-Distillation (UDCD), which integrates confidence/uncertainty into the EMA-based contrastive learning framework for medical image classification, offering a novel approach to balance the influence of transferable contrastive knowledge and supervised knowledge during self-distillation. However, similar as Reviewer4 and Reviewer8, I have some major concerns regarding the contribution of the method design and the experimental validation of its effectiveness. The claim that SCL (Spatio-temporal Contrastive Learning) is novel for this task is inaccurate, given that the CRCKD dataset also utilized SCL. Moreover, the paper lacks clarity, making it difficult to follow, and the supplementary file exceeds the page limit. Consequently, I recommend rejection.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The paper proposes a novel framework called Uncertainty-Driven Contrastive Self-Distillation (UDCD) for medical image classification, incorporating contrastive learning and EMA-based distillation with an uncertainty metric to balance knowledge transfer from teacher to student models. Strengths include the well-organized content, superior performance compared to existing methods, and innovative approach to class imbalance. However, there are several weaknesses including insufficient technical contribution, complexity in method design, lack of clarity in the method section, and inadequate visual representation. Given the strengths and weaknesses of this paper, I suggest rejecting this paper.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The paper proposes a novel framework called Uncertainty-Driven Contrastive Self-Distillation (UDCD) for medical image classification, incorporating contrastive learning and EMA-based distillation with an uncertainty metric to balance knowledge transfer from teacher to student models. Strengths include the well-organized content, superior performance compared to existing methods, and innovative approach to class imbalance. However, there are several weaknesses including insufficient technical contribution, complexity in method design, lack of clarity in the method section, and inadequate visual representation. Given the strengths and weaknesses of this paper, I suggest rejecting this paper.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The paper received mixed reviews and the criticism relates to clarity and the experimental design. This meta reviewer argues that the paper makes a valuable contribution despite its limitations. In particular, the novelty of the approach was highlighted by reviewers and ACs. The authors should improve the description and try adding requested details and improve the presentation as requested in the reviews and AC comments. Overall, the paper makes a valuable contribution and may inspire further research.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The paper received mixed reviews and the criticism relates to clarity and the experimental design. This meta reviewer argues that the paper makes a valuable contribution despite its limitations. In particular, the novelty of the approach was highlighted by reviewers and ACs. The authors should improve the description and try adding requested details and improve the presentation as requested in the reviews and AC comments. Overall, the paper makes a valuable contribution and may inspire further research.