List of Papers Browse by Subject Areas Author List
Abstract
Determining the necessity of resecting malignant polyps during colonoscopy screen is crucial for patient outcomes, yet challenging due to the time-consuming and costly nature of histopathology examination. While deep learning-based classification models have shown promise in achieving optical biopsy with endoscopic images, they often suffer from a lack of explainability. To overcome this limitation, we introduce EndoFinder, a content-based image retrieval framework to find the ‘digital twin’ polyp in the reference database given a newly detected polyp. The clinical semantics of the new polyp can be inferred referring to the matched ones. EndoFinder pioneers a polyp-aware image encoder that is pre-trained on a large polyp dataset in a self-supervised way, merging masked image modeling with contrastive learning. This results in a generic embedding space ready for different downstream clinical tasks based on image retrieval. We validate the framework on polyp re-identification and optical biopsy tasks, with extensive experiments demonstrating that EndoFinder not only achieves explainable diagnostics but also matches the performance of supervised classification models. EndoFinder’s reliance on image retrieval has the potential to support diverse downstream decision-making tasks during real-time colonoscopy procedures.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1282_paper.pdf
SharedIt Link: https://rdcu.be/dV54k
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72117-5_24
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1282_supp.pdf
Link to the Code Repository
https://github.com/ku262/EndoFinder
Link to the Dataset(s)
https://github.com/ku262/EndoFinder
BibTex
@InProceedings{Yan_EndoFinder_MICCAI2024,
author = { Yang, Ruijie and Zhu, Yan and Fu, Peiyao and Zhang, Yizhe and Wang, Zhihua and Li, Quanlin and Zhou, Pinghong and Yang, Xian and Wang, Shuo},
title = { { EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15010},
month = {October},
page = {251 -- 262}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces an EndoFinder to diagnoise polyps and explore the explainablity of deep learning methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) This paper uses self-supervised learning to learn polyp representation. 2) Using retrieval method for diagnoising polyps.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) Lots of spelling and grammatical errors, like “this results”, “image retriveal approach”. 2) The unclear definitation of the task. The main task is to classify the detected polyp. What the relations of the classification tasks with the image retrieval, re-identification tasks? 3) Different from cross-modality retrieval, the discrimination of polyps between benign and malignant is more easy. Please illustrate the advantage of the proposed method and the typical classification methods in details. 4) The combination of MAE and Contrastive learning is not novel, and they are all belong to self-supervised learning paradigm. 5）The comparison methods are not the state-of-the-art methods.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
No
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
1) The motivation of the choice of image retrieval rather than image classification method should be clearly illustrated. 2) Improving the quality of the writing.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Strong Reject — must be rejected due to major flaws (1)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1) low quality of writing 2) the unclear motivation of the method 3) the limited improvements in the classification tasks
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Reject — should be rejected, independent of rebuttal (2)
- [Post rebuttal] Please justify your decision
I maintain my point of view that the work appeared in the paper can not fully support the effectiveness of their algorithm.
Review #2
- Please describe the contribution of the paper
Introduction of EndoFinder, an online image retrieval tool that can assist on colorectal cancer diagnosis by looking for a similar and already annotated image in a large dataset
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Use of existing methodologies (CBIR) applied to a novel problem
- Good explanation of the methodology and a validation properly done
- Actual clinical use of the proposal
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- There are some doubts about the online part, regarding access to data from other institutions
- Regarding classification and considering that we deal with unbalanced datasets, per-class metrics should be included in the paper
- More histological classes should be included, including clinically relevant ones such as serrated sessile adenomas or invasive cancer
- Lack of validation against public datasets
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
No information about this that i am aware of
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The paper is clearly written and greatly structured. The methodology is sound and it has a clear clinical use. Nevertheless, there are some issues with the paper, particularly regarding methods’ validation 1) There are some public datasets (Kvasir, PolypSegmASH) that you could have compared in this paper. Why has not this been done? How does your method perform in this datasets? 2) Regarding your dataset, it only contains two classes (benign and malign) that do not represent the actual clinical needs. Moreover, data is unbalanced and, in order to understand better the performace you achive, per-class metrics shoud have been provided 3) How do you plan to deploy this in a clinical facility? Is the online part of the title something actually in the pipeline (cloud computing)? 4) Lack of details about how data in your private datasets has been acquired (manufacturer, annotations, etc.)
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Lack of proper validation, methodology is not novel but the application is.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
The rebuttal has been convincing to me, all my concerns were solved
Review #3
- Please describe the contribution of the paper
- The author introduced a self-supervised pretrained encoder to enhance the performance of polyp re-identification tasks.
- The author developed an image retrieval approach that achieved state-of-the-art performance compared to supervised classifiers.
- The author employed a hashing technique to facilitate real-time image retrieval.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The author proposed a novel self-supervised encoder designed to learn general representations from a large-scale, in-house dataset, achieving state-of-the-art performance in polyp retrieval tasks.
- The author innovatively applied hashing techniques to enhance inference speed, which is critical for clinical applications.
- The paper is well-organized, and the experiments and ablation studies are comprehensive and thorough.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper lacks sufficient details to enable reproduction of the project.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
If the author could public the code and pretrained checkpoints, it would be helpful.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Overall, it is a good paper.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The inference speed makes it meaningful for clinical use.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
Based on the author’s rebuttal, I believe that, at this moment, the proposed method holds significant potential for clinical application.
Author Feedback
We appreciate the acknowledgment from R1 and R4 regarding the technical soundness and clinical significance of our work. We will release the Polyp-Twin and Polyp-Path datasets, as well as the training codes and checkpoints of EndoFinder. Addressing concerns from R1 and R3, we aim to clarify several points.
Doubts on Classification Settings (R1): We chose a binary classification task to address the clinical need for making binary decisions (remove the polyp or leave it in situ) during colonoscopy examinations. The labels, based on histological results and revised Vienna criteria, differentiate benign from pre-malignant/malignant polyps. Vienna category 1 is considered benign, while categories 3, 4, and 5 represent pre-malignant/malignant. The Polyp-Path dataset reflects a balanced distribution (57% vs. 43%). We reported sensitivity, specificity, and F1-score to comprehensively evaluate classification performance (Table 2). While more fine-grained classification benefits precise diagnosis and is acknowledged for future work, our binary classification task validated the feasibility of our image retrieval-based framework for explainable polyp diagnosis.
Lack of Validation on Public Datasets (R1): We acknowledge R1’s suggestion but note that the Kvasir dataset lacks fine-grained histological labels, making it unsuitable for direct comparison. The PolypSegm-ASH dataset, first reported in October 2023[1], is used in the ongoing AI4PolypNET challenge[2] and was not available until 15 May 2024. Our survey of related work on page 2 indicates that few papers on polyp diagnosis release datasets, and no image datasets for polyp re-identification exist. Thus, we consider Polyp-Twin and Polyp-Path significant contributions. We would appreciate if R1 could provide more comments on the public datasets.
Questions on Online Application (R1): EndoFinder encodes images into semantic hash codes for real-time retrieval from a reference database (Figure 1). This reference database, composed of hash code and clinical semantics pairs, can be built within a single centre or multiple centres. It scales up with accumulated records, and the exclusion of original images ensures secure transfer compliant with regulations for multi-institutional exchange.
Response to R3: We believe there are significant misunderstandings regarding our work. The motivation for our image retrieval-based method is clearly described in the Introduction (page 2): “To mitigate the limitations of existing classifiers, we present EndoFinder, an image retrieval framework enhancing diagnostic explainability for colorectal polyps.” EndoFinder is built on two hypotheses: (a) self-supervised image features can retrieve the same or visually similar polyps, and (b) visually similar polyps likely share the same malignancy label. These were tested in polyp re-identification and classification tasks. Our method’s unique advantage is that classification results are accompanied by visually similar polyps in the reference database with clear histological outcomes, enhancing explainability for clinicians. Our goal was not to outperform black-box supervised classification models in accuracy but to enhance explainability. Furthermore, from a machine learning perspective, our approach is novel. While existing methods combine mask image modelling and contrastive learning, our context involves polyp images where polyps may not dominate the image. We propose a polyp-aware mask reconstruction task using polyp masks for sampling, building a polyp-aware representation that outperforms state-of-the-art CBIR methods (Table 2). To meet real-time clinical needs, we discretise output into hash codes during inference, improving retrieval speed without compromising accuracy. We appreciate the feedback on typos and will correct them in the final version. We hope our arguments are considered and the score is revised accordingly.
[1] Y Tudela et al. 2023. [2] https://pages.cvc.uab.es/ai4polypnet/?p=250
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The paper presents EndoFinder, an online image retrieval tool aiding colorectal cancer diagnosis by matching similar annotated images in a large dataset and explores deep learning methods’ explainability in polyp diagnosis. The rebuttal has effectively addressed Reviewer1’s concerns regarding public datasets and experiment settings. The major concern of Reviewer3 focuses on the accuracy improvement of this method over supervised classification methods, which is not the focus of this paper. Additionally, while the combination of MAE and contrastive learning has been explored extensively, the paper’s consideration of the unique nature of polyp masking adds innovation. Despite potential clarity issues, the paper is deemed acceptable.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The paper presents EndoFinder, an online image retrieval tool aiding colorectal cancer diagnosis by matching similar annotated images in a large dataset and explores deep learning methods’ explainability in polyp diagnosis. The rebuttal has effectively addressed Reviewer1’s concerns regarding public datasets and experiment settings. The major concern of Reviewer3 focuses on the accuracy improvement of this method over supervised classification methods, which is not the focus of this paper. Additionally, while the combination of MAE and contrastive learning has been explored extensively, the paper’s consideration of the unique nature of polyp masking adds innovation. Despite potential clarity issues, the paper is deemed acceptable.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
This paper presents EndoFinder, an online image retrieval tool to assist clinicians in the diagnosis of colorectal cancer. The paper received borderline reviews (2 weak accept and 1 reject). The authors addressed the main concerns in the rebuttal. I think the paper is valuable for the MICCAI community. It would be great to release the datasets and code as the authors commented during the rebuttal for future research.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
This paper presents EndoFinder, an online image retrieval tool to assist clinicians in the diagnosis of colorectal cancer. The paper received borderline reviews (2 weak accept and 1 reject). The authors addressed the main concerns in the rebuttal. I think the paper is valuable for the MICCAI community. It would be great to release the datasets and code as the authors commented during the rebuttal for future research.