Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Precision breast cancer (BC) risk assessment is crucial for developing individualized screening and prevention. Despite the promising potential of recent mammogram (MG) based deep learning models in predicting BC risk, they mostly overlook the “time-to-future-event” ordering among patients and exhibit limited explorations into how they track history changes in breast tissue, thereby limiting their clinical application. In this work, we propose a novel method, named OA-BreaCR, to precisely model the ordinal relationship of the time to and between BC events while incorporating longitudinal breast tissue changes in a more explainable manner. We validate our method on public EMBED and inhouse datasets, comparing with existing BC risk prediction and time prediction methods. Our ordinal learning method OA-BreaCR outperforms existing methods in both BC risk and time-to-future-event prediction tasks. Additionally, ordinal heatmap visualizations show the model’s attention over time. Our findings underscore the importance of interpretable and precise risk assessment for enhancing BC screening and prevention efforts. The code will be accessible to the public.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1554_paper.pdf

SharedIt Link: https://rdcu.be/dVZeh

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72378-0_15

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1554_supp.pdf

Link to the Code Repository

https://github.com/xinwangxinwang/OA-BreaCR

Link to the Dataset(s)

https://registry.opendata.aws/emory-breast-imaging-dataset-embed/

BibTex

@InProceedings{Wan_Ordinal_MICCAI2024,
        author = { Wang, Xin and Tan, Tao and Gao, Yuan and Marcus, Eric and Han, Luyi and Portaluri, Antonio and Zhang, Tianyu and Lu, Chunyao and Liang, Xinglong and Beets-Tan, Regina and Teuwen, Jonas and Mann, Ritse},
        title = { { Ordinal Learning: Longitudinal Attention Alignment Model for Predicting Time to Future Breast Cancer Events from Mammograms } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15001},
        month = {October},
        page = {155 -- 165}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper presents a OA-BreaCR as a new deep learning method for precise breast cancer risk assessment, integrating time-to-event order and longitudinal breast tissue changes. It outperforms existing models in predicting breast cancer risk and time-to-event outcomes based on evaluations with public and in-house datasets. The model’s interpretability is enhanced through ordinal heatmap visualizations, highlighting its potential for improving breast cancer screening and prevention efforts.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper introduces a novel method named OA-BreaCR that addresses the challenge of breast cancer risk assessment by considering the ordinal relationship of time-to-event outcomes and longitudinal changes in breast tissue. This formulation is innovative because it integrates temporal sequencing and tissue dynamics into a deep learning framework, potentially providing more accurate and clinically relevant risk predictions.
    • By validating the model on various datasets and showcasing its comparative advantages, the paper establishes the credibility and reliability of OA-BreaCR as an effective risk assessment tool.
    • OA-BreaCR has the potential to advance breast cancer screening and prevention strategies by providing clinicians with actionable insights derived from deep learning techniques, supported by clear visualization methods to the public.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The paper mentions that the code will be made accessible to the public, which is commendable for promoting transparency and reproducibility. However, the paper needs to provide sufficient methodological details and code documentation to facilitate accurate replication and implementation of OA-BreaCR by other researchers or practitioners.
    • While OA-BreaCR is evaluated on public EMBED datasets and in-house datasets, the paper does not specify the diversity and representativeness of these datasets. Further details on dataset characteristics (e.g., demographic distribution, imaging modalities) and potential biases would enhance the paper’s transparency and strengthen the generalizability of the findings.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?
    • To ensure reproducibility, the authors should specify the versions of software libraries, frameworks, and dependencies used in the implementation. Version control tools (e.g., Git) can be employed to track changes and revisions in the codebase, enabling others to reproduce the experiments under the same conditions.
    • Transparent reporting of validation and evaluation protocols is essential for assessing the robustness of the proposed method. The paper should detail the metrics used for performance evaluation, the methodology for cross-validation (if applicable), and any statistical tests employed to demonstrate significance in results.
  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    • Provide a detailed review of benchmark algorithms or state-of-the-art models in the introduction, and demonstrate how OA-BreaCR surpasses these methods in terms of accuracy, interpretability, scalability, or other relevant metrics.

    • Provide detailed information about the datasets used in the study, including demographic profiles, imaging modalities, and potential biases. Discuss how dataset diversity and representativeness were addressed to ensure the generalizability of the findings. Transparency in dataset characteristics is essential for assessing the robustness of the proposed method.

    • Consider exploring additional features, integrating multi-modal data (if applicable), or refining the model architecture to further enhance its performance and applicability in clinical settings.

    • The manuscript is will in terms of writing quality, clarity, and organization.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    -The paper introduces a novel method, which addresses an important challenge in breast cancer risk assessment by incorporating the ordinal relationship of time-to-event outcomes and longitudinal breast tissue changes. This innovative approach demonstrates a significant contribution to the field of machine learning in healthcare.

    • OA-BreaCR emphasizes clinical relevance by offering interpretable insights through ordinal heatmap visualizations, which enhance the understandability of model predictions over time. This aspect is crucial for translating complex machine learning outputs into actionable clinical interpretations, ultimately benefiting breast cancer screening and prevention efforts.

    • The paper discusses the potential impact of OA-BreaCR on advancing breast cancer risk assessment and suggests promising future directions for improvement. By acknowledging limitations and outlining next steps for research, the authors show a clear path towards enhancing the method’s performance

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper proposes to jointly model risk stratisfaction and time-to-event for breast cancer using mammograms. They use attention alignment over multi-time scans for risk prediction. They make use of an interpretable attention module, mean-var loss, probabilistic ordinal embedding.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Jointly modelling risk prediction and time-to-event, by making use of multi-time scans via an interpretable attention network is interesting and results show its robustness.

    Results indicate the proposed method performs better than baselines, which is additionally supported by attention-maps (subject to cherry-picking) and better high-risk patient identification.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    It would be good to have cross-validation results by creating multiple train-val-test splits.

    Would be interesting to see the performance of the inhouse dataset trained model evaluated (without retraining) on the public EMBED dataset, this would show the OOD performance of the model.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    In fig3, it is unclear why the number in the parenthesis - which denotes the sum of color & grey points is different for Baseline and Our method? I would expect the sum to be the same? This suggests that they are evaluated on different data?

    In the intro, it is stated that “the time-to-event estimation task could constrain the model to learn the natural ordering of the time to BC among the patients”, this statement is unsupported would be great to provide a reference or evidence for this. In my opinion, I think both tasks would supplement each other - as evident by your results?

    Defining the STP and MTP abrevations in the text would be helpful.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The inclusion of public dataset and other prior work baselines, with results showing better performance supported by ablations, heatmap and risk-distribution analysis provides confidence in the findings.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The primary focus of this manuscript is to evaluate and predict the likelihood and timing of breast cancer occurrence using sequential mammograms. Specifically, the manuscript employs ordinal learning to analyze temporal information. Moreover, it utilizes an attention mechanism to enhance the alignment in recognizing affected areas within these images, learning differences across the time series. Exceptional results were achieved through comparative experiments conducted on two datasets.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This manuscript possesses the following advantages:

    1. Unlike the majority of predictive models, this manuscript uniquely learns ordinal states and the tumor areas and their changes through both the distribution domain of features and the temporal-spatial information of images.
    2. The manuscript substantiates through extensive experimentation that ordinal states can significantly influence diagnosis, and that variations in longitudinal images can optimally predict both risk and timing.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The manuscript exhibits the following shortcomings:

    1. The readability of the Method section is poor.
    2. It is advisable to utilize standard three-line tables for the tabular presentations.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. The Method section could be enhanced by adding an overview to improve readability.
    2. Figure 1 and its descriptions in the manuscript are confusing. Figure 1 A only shows the input MG without clarifying whether it represents a single image or a sequential longitudinal image. Figure 1C merely illustrates the differential features and predictions between two images. Furthermore, although the manuscript frequently refers to multiple time points, the illustration suggests only two time points, a concept that should be clearly defined.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The methods used in the manuscript are innovative and the experimental results are substantial.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We greatly appreciate the reviewers for the effort and insightful comments regarding our submission. We are encouraged by the reviewers’ positive feedback. We will further correct errors and clarify all the concerns of the reviewers in the final version. Below are our detailed responses to the comments:

Reviewer 1. Q1. Dataset characteristics. We will include a more detailed introduction about the datasets used in our study in the revised manuscript.

Q2. Introduction. We will add a more discussion about the related work for the final version.

Q3. Further exploration. Thank you for the insightful suggestion. In future work, we will explore refining the model architecture and integrating multi-modality features to further enhance the risk prediction capability of our model. We plan to investigate the integration of additional imaging modalities, as well as non-imaging data like clinical history.

Reviewer 3. Q1. Cross-validation and OOD performance: Thank you for the suggestion. In future work, we will explore the robustness and generalizability of our risk models through cross-validation and external validation. These validations are important for assessing the applicability of our models in clinical settings.

Q2. Figure 3. We apologize for the misunderstanding. In Figure 3, we plot the distribution of the expected time to cancer for detected high-risk patients (true positive cases). Both grey and colored dots indicate true high-risk patients. The colored dots represent patients identified by the risk model with an estimated time to cancer within 5 years, while grey dots represent cases where the estimated time exceeds five years. Due to the higher performance of our proposed model, it detects more high-risk patients, leading to a difference in the sum of colored and grey points between the baseline and our method. We will clarify this explanation in the final version, ensuring that the figure is accurately interpreted.

Q3. The statement “the time-to-event estimation task could …” in the introduction. Thank you for the suggestion. We will revise the statement and add relevant references to support this claim.

Q4. The definition of STP and MTP: STP refers to single-time point methods, and MTP refers to multi-time point methods.

Reviewer 4. Q1. Readability of method: We will add more detailed explanations in the method section to improve clarity in the final manuscript.

Q2. Fig 1. Thank you for your comments. We will clarify the descriptions for Figure 1. Our proposed method is based on two-time point risk modeling. We will ensure that the multi-time point concept is clearly defined as involving two-time points.

Q3. Table. We will update the tables to a standard three-line style.



Meta-Review

Meta-review not available, early accepted paper.



back to top