Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Panoramic image segmentation in computational pathology presents a remarkable challenge due to the morphologically complex and variably scaled anatomy. For instance, the intricate organization in kidney pathology spans multiple layers, from regions like the cortex and medulla to functional units such as glomeruli, tubules, and vessels, down to various cell types. In this paper, we propose a novel Hierarchical Adaptive Taxonomy Segmentation (HATs) method, which is designed to thoroughly segment panoramic views of kidney structures by leveraging detailed anatomical insights. Our approach entails (1) the innovative HATS technique which translates spatial relationships among 15 distinct object classes into a versatile “plug-and-play” loss function that spans across regions, functional units, and cells, (2) the incorporation of anatomical hierarchies and scale considerations into a unified simple matrix representation for all panoramic entities, (3) the adoption of the latest AI foundation model (EfficientSAM) as a feature extraction tool to boost the model’s adaptability, yet eliminating the need for manual prompt generation in conventional segment anything model (SAM). Experimental findings demonstrate that the HATS method offers an efficient and effective strategy for integrating clinical insights and imaging precedents into a unified segmentation model across more than 15 categories. The official implementation is publicly available at https://github.com/hrlblab/HATs.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1451_paper.pdf

SharedIt Link: https://rdcu.be/dY6is

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72083-3_15

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1451_supp.pdf

Link to the Code Repository

https://github.com/hrlblab/HATs

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Den_HATs_MICCAI2024,
        author = { Deng, Ruining and Liu, Quan and Cui, Can and Yao, Tianyuan and Xiong, Juming and Bao, Shunxing and Li, Hao and Yin, Mengmeng and Wang, Yu and Zhao, Shilin and Tang, Yucheng and Yang, Haichun and Huo, Yuankai},
        title = { { HATs: Hierarchical Adaptive Taxonomy Segmentation for Panoramic Pathology Image Analysis } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15004},
        month = {October},
        page = {155 -- 166}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The goal of this paper is to segment 15 various structures in kidney pathology whole slide images. To do so, the authors propose a hierarchical scale matrix – the hierarchical scale matrix contains scale ratio between structures. A novel loss term derived from the hierarchical scale matrix is suggested. In addition, the authors employ a token-based dynamic EfficientSAM.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The idea of a hierarchical scale matrix derived from areas of different structures is novel. This would form relationship between kidney structures in different scales which may potentially lead to more accurate inference.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    My main concern is evaluation. To really understand the benefit of a hierarchical scale matrix, PrPSeg [7] should have been compared in each class in Table 2. According to Table 3, the proposed method shows marginal improvements from PrPSeg. (1.48% improvement in regions, 0.36% in units, 0.13% in cells, 0.69% in average)

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    (1) As mentioned in 6, PrPSeg [7] needs to be included in Table 2 to show the benefit of a hierarchical scale matrix. (2) Reference [7] must be cited in Section 2.1 because a hierarchical taxonomy matrix has been introduced in [7] as universal proposition matrix. (3) Detailed information about the data in Section 3 is needed. For example, how many cases are in the dataset? Is this a public dataset or an internal dataset?

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Reject — should be rejected, independent of rebuttal (2)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The main novelty is in a hierarchical scale matrix but results do not clearly demonstrate its effect. According to Table 3, its contribution seems to be marginal.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • [Post rebuttal] Please justify your decision

    I appreciate the authors for many clarifications. It seems the main novelty of the current work comparing to PrPSeg is that the current work can segment 15 classes using d-EfficientSAM and HSM. In this case, this point should have been clearly highlighted in Introduction – specifically, (1) why 15-class segmentation is really needed in renal pathology (introducing some clinical applications) and (2) why the previous PrPSeg was not possible to perform 15-class segmentation. To demonstrate my second point, I still believe results from PrPSeg should be included in Table 2. If Table 2 will be modified, I am willing to increase my score, but I am still leaning towards rejection. Rank of this paper in my stack of rebuttal papers: 2/2.



Review #2

  • Please describe the contribution of the paper

    This paper proposed a method to systematically incorporate clinically important tissue analytical information into segmentation of renal pathology images. To this end, the authors developed multi-scale relationships between various anatomical structures that encodes their spatial relationship and scale relationships. In addition, the authors developed efficientSAM with class and scale related token prompts for segmentation modeling and inference. The evaluation on a custom dataset demonstrated superior performance over a set of baseline models.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The proposed method provides a systematic approach for incorporating clinically relevant information to renal pathology image segmentation and potentially this idea can be extended to other pathology settings.
    • The writing is clear and the Figure illustrations are well designed.
    • A comprehensive list of SOTA and commonly used models in biomedical image segmentation were included in evaluation.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The level of novelty seems a bit concerning. The proposed method leveraged the design in Prpseg (citation 7 in the manuscript) but added the design of hierarchical scale matrix (HSM) on top of the Prpseg.
    • It seems not as evident as stated in the manuscript about the effectiveness of hierarchical scale matrix (HSM). Looking at Table 2, the performance of proposed HSM seemed to increase performance to a small extent when compared with Prpseg (citation 7 in the manuscript). In addition, when comparing with SOTA, Prpseg was included in Table 3 but not included in Table 2.
    • Details of image and labels are missing, such as the number of patients, number of instances annotated in each patch or in total (since the authors mentioned that they used non-exhaustive annotations).
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    A custom dataset was used in this study of unknown patient number and instance number for each object type, which seems not easy to reproduce due to unknown annotation protocols and possibly extensive annotation efforts needed.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    • Please refer to the “weakness” section for major concerns.
    • The evidence for the effectiveness of the proposed hierarchical scale matrix (HSM) may be strengthened by providing object type level comparison with the Prpseg (citation 7 in the manuscript), like the comparisons with other baselines in Table 1. In addition, experimental repeats with different random seeds and calculation of standard deviation will be helpful for demonstrating the superior performance of HSM.
    • Since the authors emphasized the usage of clinically relevant information for model designs, can the authors add discussion of how the proposed panoramic segmentation can potential be leveraged in clinical applications? May there be barriers (such as the extensive annotations of all 15 types of objects? Is it and when is it necessary to annotate all 15 types?) to practical applications?

    Minor:

    • Typo: Page 5 - “Instead of … a pre-difined learnable token bank” “pre-difined” -> pre-defined
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall this is a well-written and well-motivated study with comprehensive evaluation. A few concerns lowered my scoring: (1) level of novelty (2) missing detailed comparison with a recent SOTA, which the authors’ design built upon.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper introduces a novel method for segmenting panoramic views of kidney structures in computational pathology. The proposed method leverages detailed anatomical insights to address the challenges posed by the morphologically complex and variably scaled anatomy of kidney pathology. It adopts th latest AI foundational model EfficientSAM as feature extractor. Experimental analysis reveal unified segmentation model across 15 categories.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper addresss the panoramic image segmentation in computational pathology by leveraging detailed anatomical insights, covering multiple layers and structures ranging from regions to cells.
    2. Paper presented hierarchical taxonomy matrix and a hierarchical scale matrix allowing segmentation across multiple object classes.
    3. This paper integrates the foundational models with token bank favoring weak prompts over pixel-wise prompts.
    4. Evaluated on more than 15 categories demonstrates effectiveness of the model.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The paper’s reliance on anatomical specificity poses a significant challenge for its translation into clinical settings. Without comprehensive evaluation across diverse anatomies on a large scale, the model’s applicability may remain restricted.
    2. PrPSeg employ a residual U-Net backbone and incorporate class-aware and scale-aware tokens, the distinction between the two methodologies, apart from their use of EfficientSAM, lacks clarity. Notably, PrPSeg utilizes 8 classes, whereas HATs extends this to 15.
    3. The paper lacks details regarding the dataset’s accessibility if its public or institutional proprietary. If the dataset is indeed publicly available. Also, were the weak annotation manually created is unclear.
    4. While reading the paper hints at limitations concerning generalizability and the human annotations, these constraints are not sufficiently elaborated (if this are really contraint is bit unclear).
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Author can use Anonymous GitHub to provide code (Ref: https://anonymous.4open.science/) if possible.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. Evaluation of generalizability on more diverse dataset digitized with different protocols.
    2. Foundational model such as UNI (https://www.nature.com/articles/s41591-024-02857-3) specific to computational pathology are available, it will be interesting to see the results with this foundational models.
    3. Explore methods to improve the identification and refinement of the 15 object classes, possibly through advanced techniques or is it only possible through user-driven annotations?
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper introduces HATs, a method for segmenting kidney structures in pathology images. It utilizes a hierarchical taxonomy matrix and scale matrix to incorporate anatomical relationships and scale knowledge. The method integrates a dynamic EfficientSAM network architecture with weak token prompts for efficient segmentation. Experimental results demonstrate superior performance in segmenting anatomical structures across 15 categories, contributing to more accurate diagnosis and treatment efficacy in renal pathology.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We appreciate all the reviewers for their insightful comments and have summarized them into three major and two minor concerns.

[Major concerns] C1. Clarification of novelty and benefit of the HATs beyond PrPSeg (R1, R3, R5): – We apologize for the lack of clarity and would like to highlight our contributions in this rebuttal: (1) New dynamic EfficientSAM Backbone: We introduce a novel token-based dynamic EfficientSAM (d-EfficientSAM) backbone with enhanced multi-scale capability, achieving superior semantic segmentation for 15-class segmentation in both PrPSeg and HATs. (2) Quantitative modeling of anatomical relationship beyond a binary matrix: The HATs model captures the relative relationships between multiple classes using a new hierarchical scale matrix (HSM). This model transitions PrPSeg’s hierarchical taxonomy matrix (HTM) from a binary relationship to a fully quantitative relationship. (3) More comprehensive setting in panoramic kidney pathological segmentation: We have doubled the complexity of panoramic multi-scale kidney pathological segmentation from 8 classes to 15 classes, allowing a more comprehensive pathological panoramic segmentation using a single model. (4) Open-sourcing the HATs model: We will open-source the HATs model to promote more reproducible research, unlike PrPSeg, which was not open-sourced.

C2. Performance gain from PrPSeg to HATs (R1, R3, R5): – The PrPSeg model implemented in Table 3 features our new d-EfficientSAM backbone. To avoid confusion, we will refer to this model as “PrPSeg (d-EfficientSAM)” in the manuscript. For clarity, we have also implemented the original PrPSeg with a CNN-based backbone, which has a performance of dice 65.94. Compared to the original PrPSeg (CNN), our HATs model outperforms it by 1.64%, demonstrating the added benefit of introducing d-EfficientSAM beyond HSM. – The Wilcoxon signed-rank test shows that the improvements from HATs method with PrPSeg (CNN) and PrPSeg (d-EfficientSAM) are statistically significant (p-value < 0.001) in both Table 2 and 3.

C3. More comprehensive assessments on HATs (R1, R3, R5): – As suggested by R1, we have repeated experiments with different random seeds with the settings from the ablation study in 8-class segmentation. According to the results (backbone only: dice 74.38, std 0.097, HTM: dice 75.12, std 0.085, HTM+HSM: dice 75.44, std 0.045), the proposed method consistently achieved better performance. – Additionally, to address R3’s question, we will provide results from three ablation study settings in Table 3 using the PrPSeg (CNN) backbone (backbone only: dice 64.95, HTM: dice 65.94, and HTM+HSM: dice 66.65, p-value < 0.001 compared with HATs).

[Minor concerns] C4. Motivation of the HATs for clinical application (R1, R5): – Recently, pathomics has introduced a fully quantitative approach to enhancing the current semi-quantitative clinical guidance, enabling the development of fully quantitative biomarkers. Detailed segmentation across multiple organs is essential to achieve these pathomics biomarkers. We hope that this model will facilitate comprehensive multi-scale, multi-object segmentation, leading to more thorough pathomics biomarkers for kidney pathology.

C5. Typos, detailed data information, discussion and PrPSeg illustration (R1, R3, R5): – We appreciate all of the suggestions and comments from reviewers. We will correct all of the typos and add detailed data introduction, PrPSeg illustration, and motivation discussion as suggested by the reviewers in the final version of the manuscript.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    NA

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    NA



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    This paper caused some issues with overlapping paper accepted at CVPR’24 which inflated some contributions. Gladly, the rebuttal clarified the outstanding issues and hence, is acceptable for publication.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    This paper caused some issues with overlapping paper accepted at CVPR’24 which inflated some contributions. Gladly, the rebuttal clarified the outstanding issues and hence, is acceptable for publication.



back to top