Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Survival analysis is critical for clinical decision-making and prognosis in breast cancer treatment. Recent multimodal approaches leverage histopathology images and bulk RNA-seq to improve survival prediction performance, but these approaches fail to explore spatial distribution at the cellular level. In this work, we present a multimodal hypergraph neural network for survival analysis (MHNN-surv) that introduces a pre-trained model for spatial transcriptomic prediction. The method is characterized by making full use of histopathological images to reveal both morphological and genetic information, thus improving the interpretation of heterogeneity. Specifically, MHNN-surv first slices Whole-Slide Imaging (WSI) into patch images, followed by extracting image features and predicting spatial transcriptomic, respectively. Subsequently, an image-based hypergraph is constructed based on three-dimensional nearest-neighbor relationships, while a gene-based hypergraph is formed based on gene expression similarity. By fusing the dual hypergraphs, MHNN-surv performs an in-depth survival analysis on breast cancer using the Cox proportional hazards model. The experimental results demonstrate that MHNN-surv outperforms the state-of-the-art multimodal models in survival analysis.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/4136_paper.pdf

SharedIt Link: https://rdcu.be/dV1WF

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72384-1_43

Supplementary Material: N/A

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Cai_Survival_MICCAI2024,
        author = { Cai, Shangyan and Huang, Weitian and Yi, Weiting and Zhang, Bin and Liao, Yi and Wang, Qiu and Cai, Hongmin and Chen, Luonan and Su, Weifeng},
        title = { { Survival analysis of histopathological image based on a pretrained hypergraph model of spatial transcriptomics data } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15003},
        month = {October},
        page = {455 -- 466}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposed a survival prediction method using WSI and genetic information. It uses hypergraphs to encode the WSI image patches and the spatial genetic information, respectively. By fusing these two kinds of hypergraphs, survival risk can be predicted by a Cox model. Experimental results show advantage of the proposed method over existing methods.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    -use hypergraphs to represent spatial genetic information and also unify the representations of WSI and genetic information.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    -the description of gene-based hypergraph is too coarse. -how the accuracy of the IGI-DL model affects the hypergraph and the final prediction accuracy is not mentioned.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    -description of section 3.2 (gene-based hypergraph construction) and 3.3 needs to be improved. -evaluation of IGI-DL in gene-based hypergraph construction should be added in the experiment.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    -use hypergraphs to achieve spatial gene representation. -unify the representation of WSI and genetic information.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    In this work, the authors propose a novel multi-modal hypergraph neural network to capture the diverse relations present in the different modalities of genomics and histopathology as different sets of hyper edges on the hypergraph. Utilising this constructed hypergraph as input to a hypergraph neural network, the authors propose to predict survival using a Cox-based loss function.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper proposes an interesting and novel approach to tackling the multiple types of relations present in multi-modality spatial data by leveraging hypergraphs. This is interesting in that such an idea can be easily extended to work with multiple modalities by constructing more hyperedges for the input hypergraph and can also be applied in other spatial-data settings.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The particular approach taken in the paper relies on utilising histopathology information to predict the spatial gene expression before feeding the resulting spatial gene expressions and histopathology information to the proposed method. The validity of the predicted spatial gene expressions is unclear, and the methodological details of this component are missing.
    • The method is evaluated only on one dataset, and thus the applicability of the pipeline to other settings/datasets remains to be seen.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Below, please find specific comments about the manuscript.

    In the abstract:

    • “slicing the WSI to patch the image set” is unclear. Do you mean that the WSI is divided into non-overlapping patches?
    • What is the “three-dimensional nearest neighbor relationship” being mentioned here? Since the data is naturally in 2D, such a sentence without further information is confusing.

    In Section 3.2

    • “Image-based hypergraph construction” Why do you define a third dimension which takes into account all three R, G, B components instead of simply using R, G, B values directly? Or using one value for each R, G, B (based on normalisation of variances etc.) to get a 5-D vector instead? What was the rationale?
    • “Gene-based hypergraph construction” what is the IGI-DL model used to predict the gene expressions? What is the input to this model? What does the model do? Is the reference [24] correct?
    • How are the features of the different modalities aggregated at the node-level for the multimodal hypergraph construction? Are the features standardised/normalised to ensure uniform scales across the modalities?
    • The proposed approach uses only one pass of vertex -> hyperedge information in the hypergraph network. What happens if you instead use hypergraph networks which perform two-way updates, such as HGSurvNet [12]?

    In Section 4, Experiments and Results:

    • The C-index value can be in the range 0-0.5 also. It would just mean that the ordering is exactly opposite to what is desired..
    • Can you comment on the hypergraph components important for the survival prediction in some form of an explainability experiment?
    • Even if the focus of one of the previous tools is only on breast cancer data - are there other breast cancer pathology datasets to test on for experimental comparisons?
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper proposes an interesting approach to leverage hypergraphs to perform a multi-modality aggregation of spatial data. However, important details of the histology -> gene prediction pipeline are missing. Further, since the model is evaluated only on one dataset, it is unclear how generalisable the proposed method is.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The authors developed a novel model to integrate whole slide pathology images, gene expression and spatial information in accounting for cancer prognosis. They use a fused graph neural network to predict survival rate showing an improvement of up to 0.065 in C-index.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Proposed method shows improvement compared to existing approaches in breast cancer prognostic performance.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    This approach seems to give relatively modest improvements in survival prediction. Additionally although effective, it seems to lack interpretability - which areas show what gene expression.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Link to some of the source source code provided.

    Data is publicly accessible.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Well-written with clear logic, tables, and figures.

    I do not find the comparisons in figure 3 showing improved performance with incorporation of both whole slide imaging and genetic data particularly persuasive as I would expect performance to improve with additional data. Is this statistically robust, how does the performance change when the networks are retrained with different seeds or k-fold validation?

    While the paper discusses a spatial transcriptomic approach, the applied method predicts transcription per WSI subpath of size 224x224 and considers 69 genes which I would not consider ‘-omic’ scale. I recommend making this distinction clear throughout the manuscript.

    Since the transcriptomic data is inferred by another algorithm, have there been any study in the accuracy of this algorithm on datasets beyond its internal test and validation cohort? Furthermore do its predictions for the current paper WSI-patches line up with human interpretation?

    Finally only patches with at least 0.7 of tissue were analyzed, discarding edge WSI patches. Is there performance increase if the 0.7 freehold is decreased or increased, furthermore can the authors explain if edges of WSI contain additional information not represented in internal WSI patches?

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I find this to be a subtle improvement on existing approaches.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

N/A



Meta-Review

Meta-review not available, early accepted paper.



back to top