List of Papers Browse by Subject Areas Author List
Abstract
Selecting regions of interest (ROIs) in whole-slide histology images (WSIs) is a crucial step for spatial molecular profiling. As a general practice, pathologists manually select ROIs within each WSI based on morphological tumor markers to guide spatial profiling, which can be inconsistent and subjective. To enhance reproducibility and avoid inter-pathologist variability, we introduce a novel immune-guided end-to-end pipeline to automate the ROI selection in multiplex immunofluorescence (mIF) WSIs stained with three cell markers (Syto13, CD45, PanCK). First, we estimate immune infiltration (CD45+ expression) scores at the grid level in each WSI. Then, we incorporate the Pathology Language and Image Pre-Training (PLIP) foundational model to extract features from each grid and further select a subset of grids representative of the whole slide that comparatively matches pathologists’ assessment. Further, we implement state-of-the-art detection models for ROI detection in each grid, incorporating learning from pathologists’ ROI selection. Our study shows a significant correlation between our automated method and pathologists’ ROI selection across five different types of carcinomas, as evidenced by a significant Spearman’s correlation coefficient (> 0.785, p < 0.001), substantial inter-rater agreement (Cohen’s kappa > 0.671), and the ability to replicate the ROI selection made by independent pathologists with excellent average performance (0.968 precision and 0.991 mean average precision at a 0.5 intersection-over-union). By minimizing manual intervention, our solution provides a flexible framework that potentially adapts to various markers, thus enhancing the efficiency and accuracy of digital pathology analyses.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/3757_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/3757_supp.pdf
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Gau_Immuneguided_MICCAI2024,
author = { Gautam, Tanishq and Gonzalez, Karina P. and Salvatierra, Maria E. and Serrano, Alejandra and Chen, Pingjun and Pan, Xiaoxi and Shokrollahi, Yasin and Ranjbar, Sara and Rodriguez, Leticia and Team, Patient Mosaic and Solis-Soto, Luisa and Yuan, Yinyin and Castillo, Simon P.},
title = { { Immune-guided AI for Reproducible Regions of Interest Selection in Multiplex Immunofluorescence Pathology Imaging } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15004},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a pipeline to select ROIs based on multiplex immunfluoresce (mIF) WSIs for spatial molecular profiling. The pipeline consists of three major steps: i) Automated Immune Scoring based on quantification on different channels of fluorescence images. ii) Selection of grids containing ROIs that matches reference grids (annotated by pathologists) in the feature space using PLIP model iii) a finely tuned object detection to generate the final ROIs. The pipeline generated ROIs are compared to manually selected ones and achieve an averaged IOU about 90%.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.ROI selection for spatial molecular profiling is an important and biological-relevant.
- The proposed pipeline seems to achieve good performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper is not innovative with respect to AI or machine learning.
- I am not sure why PLIP model is used here. As far as I know, the training data of PLIP model is also mostly H&E stained images. It is not clear whether PLIP model is suitable for the fluorescence multi-flexing images here and the feature space of PLIP model is informative.
- The evaluation is purely based on expert-selected ROIs. If the ultimate downstream task is for molecular profiling. Is the AI-selected/Manuel-select ROIs making a significant difference on the final molecular profilers? Which are more valid/better?
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
I feel the paper is difficult to reproduce the results reported in the paper if the dataset and annotation are not released, although the method itself is not complicated and can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Authors should justfiy the use of PLIP as a feature extractor. Moreover, authors should perform down-stream tasks to prove that the automatically selected ROIs leads to similar molecular profiling as the existing protocol based on manual selection.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Reject — should be rejected, independent of rebuttal (2)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Small Novelty in Methodology and relatively weak evaluation.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
The authors well explains a few key concerns of my original assessment. Whilst I still think the novelty of the paper is not particularly strong in terms of computational/mathematical methodology, it is an important research field and maybe worth to improve its visibility in miccai community
Review #2
- Please describe the contribution of the paper
In pathology, the selection of ROIs in whole-slide images is important for downstream procedures such as molecular scoring, diagnostic assessments, biomarker identification, etc. Traditionally, most of the work are done manually by pathologist, which is tedious and subjective. This paper propose an automatic ROI selection pipeline for multiplex immunofluorescence (mIF) WSIs. Evaluation of the proposed pipeline show that the selection results are highly correlated with pathologists’ view, indicating the correctness of the pipeline. Compared to H&E pathology images, mIF contains richer information, thus are more challenging to perform analysis on. Thus, this ROI selection pipeline can help minimize manual intervention and improve efficiency.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main stengths of the paper are as follows:
- The paper is well-written and easy to follow. Particularly, the method part is clear, which includes the scoring stage, the classification and similarity matching stage, and the ROI selection stage. The topic is of sufficient interest to the MICCAI audiences and the pathology community.
- I like the evaluation part. The authors first validate the correctness of the automated immune scoring by comparing to pathologists’ scores. Similarly, they also assess the quality of gird selection and then the quality of ROI selection. Since cancer research is a high-stake field, in order to really make AI works under real clinical settings, it is important to have human involved during the development of pipeline to ensure the correctness.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses of the paper are as follows:
- The authors did not provide enough experiment details, such as what hyper-parameters are involved in each stage and how did the authors select them. Training details are also missing.
- Given that H&E WSIs are more popular at least for now, does the proposed pipeline works on H&E WSIs?
- It seems that PLIP is trained on H&E WSIs, how did the authors adapt it to mIF WSIs? This needs further clarifications.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
It would be great if the authors can make the dataset publicly available for the research community to follow this direction.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Please comment on the weaknesses mentioned above during the rebuttal period.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Based on my evaluation above, I believe the strengths of this paper outweigh its weaknesses. Therefore, I recommend weak accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
After reading the rebuttal and other reviews, I maintain my original score.
Review #3
- Please describe the contribution of the paper
The authors propose a multi-step approach for automating Regions of Interest (ROI) Selection in Multiplex Immunofluorescence (mIF) Pathology Imaging. The approach consists of automated immune scoring (aIS) for a 3000x3000 pixels grid cell, feature extraction via PLIP, and final 900x900 pixels ROI detection within a larger grid cell via YOLO-v8.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors propose Automated Immune Scoring method and utilise SOTA approaches the later stages of their pipeline (PLIP, YOLO-v8). Strong performance on the test set and high potential clinical value.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The results rely on powerful building blocks used by the authors: PLIP and YOLO-v8, which account for 2 out of the 3 components in the pipeline.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The algorithm is described well enough to reproduce it. However, the authors used a private dataset with multiple stages of annotation. So to reproduce the algorithm on any other dataset, one would need to replicate the annotation steps first.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Section 2.3: please clarify if black, yellow, and green are universally used colours or if you chose them for visualisation.
Section 2.4, first paragraph: please explain if it’s usual not to have any samples in aIS > 50% group:“Our tissue samples did not have grids with aIS > 50%”
Section 2.4: Selecting similar regions based on a cosine similarity of an embedding from a pre-trained model is a standard practice for retrieval in the natural imaging domain. There is a number of works in histopathology domain too, some are used for mining datasets with many WSIs for similar slides, other - for finding similar ROIs on WSIs, e.g. for enriching annotations of underrepresent classes. Consider referencing some of the recent works.
Section 2.4: can you explain the choice of PLIP over other histopathology models?
Figure 2: Consider changing the visualisation of Spearman’s Correlation and Cohen’s kappa scores. In a single heat map, they resemble a confusing matrix, with missing diagonal values. They can be reported as 2 sets of 3 values in each.
Figure 3, (b): does the pink/purple border signify anything?
Section 3.1, last paragraph: I thought there were no samples with score in the “> 50%” group. Please clarify.
Section 3.3: consider reminding the readers that TIL is a tumor-infiltrating lymphocyte. I had to find where it was first mentioned.
Please order the cited references in ascending order to make look-up easier, e.g. page 2 [19, 2, 5] -> [2, 5, 19]. I think it’s easier to look up the references when they are sorted.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Strong results, well written, good methodology.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
The authors addressed my questions in the rebuttal. I believe that this paper will be of interest to our community.
Author Feedback
Dear Reviewers, Area Chairs and Program Chairs, We thank the constructive feedback provided by all the reviewers on our application manuscript. In this rebuttal, we group and address the major critiques that the review process detected and provide clarification on decisions previously unclear in the text.
We thank Reviewers #1 and #3 for their observations and questions regarding using the PLIP foundational model in analyzing multiplex immunofluorescence (mIF) images. Closely relevant research conducted by Yu et al. (“A Multi-Granularity Approach to Similarity Search in Multiplexed Immunofluorescence Images”, 2023) supports using PLIP for feature extraction and similarity search in mIF images. Regarding the concerns about transfer learning, to validate our selection of PLIP, we used PCA and t-SNE to study the feature embeddings extracted by PLIP from mIF images, which showed distinct clustering patterns of the image embeddings. The dispersion along the principal components suggested that PLIP can capture significant variance and underlying structures within the mIF images, indicating that the abstract features learned from H&E images (e.g., cellular morphology and tissue architecture) are transferable to mIF images. We did not include these details in the original submission due to the 8-page limit and considered that the validation of PLIP on mIF images is not the main goal of our research.
Regarding the applicability to H&E-stained Whole Slide Images (WSIs) mentioned by Reviewer #3, while our current pipeline is optimized for mIF, it contains flexible components that can be adapted to different staining techniques. These elements include pre-processing stages that can be tailored to handle the specific characteristics of H&E staining, thereby extending the utility of our workflow to a broader range of pathological analyses. Moreover, the PLIP model’s existing proficiency with H&E WSIs complements this flexibility, ensuring our methodology remains versatile and broadly applicable.
As to the validation of the Regions of Interest (ROIs) selection (Reviewer #1), we agree with the reviewer that the end goal is molecular profiling. However, without a clear optimization goal, the process can result with many candidate ROIs. Before digging into such complexity, we chose to start simple and retrospectively assess the pan-cancer automation of the process. Generating an AI-based spatial molecular profiling is within our overarching goals, but it requires more time for molecular assays, and it is outside the specific scope of this manuscript. Our study is designed to use expert-selected ROIs as a benchmark to learn from pan-cancer and clinically relevant data, enhancing its applicability. We show that AI-selected ROIs closely match pathologist’s selections in terms of consistency and coverage, this being a crucial step in molecular profiling, where precise ROI selection impacts the detection and quantification of biomarkers.
On the specifics of the training details (Reviewer #3), we used a YOLOv8-S model with a batch size =64, an image size of 1280 pixels, learning rate =0.01, momentum =0.937, and a weight decay = 0.0005 for 1000 epochs with a patience=200. After multiple runs with different values, we achieved the following training scores: box loss of 0.523, class loss of 0.399, and distribution focus loss of 1.001. The validation scores were box loss =1.686, class loss =0.985, and distribution focus loss =1.51. The images were tested at multiple magnifications (20x, 10x, and 5x), with 20x achieving the best results in the main submission.
After approval, these changes can be added to the main submission if needed. By addressing these points, we aim to clarify our application, underscore its robustness by applying SOTA models within a rich dataset, and highlight the potential of our study.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The authors have adequately addressed all the concerns.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The authors have adequately addressed all the concerns.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A