List of Papers Browse by Subject Areas Author List
Abstract
Longitudinal imaging examinations are vital for predicting pathological complete response (pCR) to neoadjuvant therapy (NAT) by assessing changes in tumor size and density. However, quite-often the imaging modalities at different time points during NAT may differ from patients, hindering comprehensive treatment response estimation when utilizing multi-modal information. This may result in underestimation or overestimation of disease status. Also, existing longitudinal image generation models mainly rely on raw-pixel inputs while less exploring in the integration with practical longitudinal radiology reports, which can convey valuable temporal content on disease remission or progression. Further, extracting textual-aligned dynamic information from longitudinal images poses a challenge. To address these issues, we propose a longitudinal image-report alignment-guided model for longitudinal mammogram generation using cross-modality radiology reports. We utilize generated mammograms to compensate for absent mammograms in our pCR prediction pipeline. Our experimental result achieves comparable performance to the theoretical upper bound, therefore providing a potential 3-month window for therapeutic replacement. The code will be accessible to the public.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1552_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1552_supp.pdf
Link to the Code Repository
https://github.com/yawwG/LIMRA/
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Gao_Improving_MICCAI2024,
author = { Gao, Yuan and Zhou, Hong-Yu and Wang, Xin and Zhang, Tianyu and Han, Luyi and Lu, Chunyao and Liang, Xinglong and Teuwen, Jonas and Beets-Tan, Regina and Tan, Tao and Mann, Ritse},
title = { { Improving Neoadjuvant Therapy Response Prediction by Integrating Longitudinal Mammogram Generation with Cross-Modal Radiological Reports: A Vision-Language Alignment-guided Model } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15001},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper studies the issue of lacking imaging in the context of the prediction of therapy response for breast cancer treatment. More specifically, the authors propose a method of mammogram generation relying on clinical reports from MRI studies. The approach is technically appealing allowing to generate realistically-looking imaging on the text data only.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Overall the paper is clear and exposes well the proposed method. The technological aspect is appealing, as exposes the potential of generating plausibly looking imaging data from textual reports. The provided drawings, illustrations, and numerical results help in understanding the outcomes of the proposed method. The statistical significance evaluation with p-values is a good reflex.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness of the paper is the uncertainty about the clinical relevance. While there is a technical interest in the multi-modality approach, generating imaging data from texts, it does not clearly stand out how the generated data could be used. That is, the generated mammogram as a means for therapy effectiveness assessment may be error-prone. I would like the authors to more clearly position the limitation and explain motivation.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
No
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
I have a few comments: - [*] Could the authors provide more discussion about the relevance and interest of mammogram generation as a means for pCR prediction? - [Method] The authors propose a network capable of generating mammograms from a mammography image, a clinical report, and pathological information. Could the authors clarify how the pathological information (C) is integrated? - [Method] Moreover, could the authors comment on the use of the MRI in the processing and their decision not to use MRI as additional input? - [Experiments] Could the authors provide more details about the generated images? What is resolution? - [Experiments] As the authors provide visual results, I wonder whether there are any numerical results that could be added, i.e., dice score, IoU, or any lesion size metrics with MAE or MSE? Could the authors comment?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Generally the paper appears to be a good conference material, with clearly exposed methods, structured experiences, and detailed metrics. The clinical relevance is a question mark but does not prevents from a weak accept.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The manuscript employs a two-stage framework. Initially, post-NAT images are generated from pre-NAT X-ray images through a longitudinal image report alignment method. Subsequently, a multimodal prediction of the pCR stage is conducted using the generated post-NAT X-ray images. The experimental results validate that the generated pCR images achieve outcomes comparable to those of the original images, which is above the threshold for specific clinical applications. This provides a potential three-month therapeutic alternative window.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The manuscript presents several technical and clinical advantages as follows:
- Compared to traditional generative methods, the manuscript employs a CLIP approach for aligning image reports, which more effectively reflects the changes in tumor images before and after NAT. This provides more realistic and reliable X-ray images for downstream tasks, such as pCR detection.
- The validation of pCR using the generated images achieves performance equivalent to that of using post-NAT images directly, which can significantly enhance the work efficiency of physicians.
- The manuscript conducts extensive and reliable analyses to validate both the generated images and the pCR results. It also provides a thorough explanation of potential issues, as exemplified by the rationale behind the choice of evaluation metrics (e.g., why LPIPS was used instead of SSIM and PSNR).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The manuscript has the following limitations:
- In the evaluation phase, the manuscript solely utilizes the generated post-NAT images for prediction without considering the relationship between the changes before and after NAT. There is currently ongoing work in this direction that addresses this aspect.
- The manuscript does not explicitly outline how multi-view (CC/ELO or other) pCR prediction experiments are conducted for each patient. It remains unclear whether all images are used directly for prediction or if alternative strategies are employed.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Do you have any additional comments regarding the paper’s reproducibility?
The manuscript states that the reports used were from MRI, yet it does not explain why MRI images were not used for PCR prediction. It is possible that the authors have already conducted experiments in this regard but did not obtain satisfactory results. Although the manuscript’s intention is to avoid the invasive and radiation-associated MRI examinations
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- The authors should elaborate on the details of pCR prediction, such as the multi-view training and inference issues previously mentioned.
- It is recommended that the authors include MRI results in the manuscript or a supplementary document. If MRI results cannot be provided, the authors should provide a detailed explanation of the reasons, such as issues with data acquisition during the MRI process.
- The manuscript should incorporate hyperlinks to facilitate easier access and improve the reading experience for the readers.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The framework presented in this manuscript is quite novel and has been thoroughly validated through extensive experiments, which holds significant clinical relevance. Timely pCR assessment using post-NAT images is also a challenging issue in clinical practice. There are only a few minor details that require attention.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper presents the longitudinal image-report alignment (LIMRA) method, a novel approach for generating longitudinal mammograms from cross-modal radiological reports. By integrating LIMRA into a predictive framework for early neoadjuvant therapy (NAT) response in breast cancer patients, the study indicates the potential for timely and accurate treatment response assessments. Achieving performance comparable to the upper bound, the method suggests a promising 3-month window for therapeutic replacement. Furthermore, the paper sets the stage for expanding these concepts to other breast imaging modalities, thereby improving clinical decision-making in breast cancer management.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The paper highlights the practical implications of its findings, suggesting a potential 3-month window for therapeutic replacement based on the predictive capabilities of the model. This insight has significant implications for treatment planning and patient management in breast cancer care. -The study rigorously evaluates the performance of the proposed method, demonstrating comparable results to the upper bound. This indicates the effectiveness of the approach in predicting NAT response and suggests its potential utility in clinical practice.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-The paper may not adequately address assumptions or limitations associated with the proposed method, such as the assumption of uniform response patterns across patients or the potential impact of confounding factors on predictive performance.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Expand the evaluation of the proposed method beyond comparison with an upper bound. Include a thorough analysis of its performance under different scenarios, such as varying dataset sizes, imaging modalities, or treatment regimens. Utilize a broader range of evaluation metrics to provide a comprehensive assessment of the method’s strengths and weaknesses.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The study rigorously evaluates the performance of the proposed method and demonstrates its effectiveness in predicting NAT response. Achieving comparable results to the upper bound suggests the method’s robustness and potential utility in clinical practice.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We greatly appreciate the reviewers for the effort and insightful comments regarding our submission. We are encouraged by the reviewers’ positive feedback in terms of technical novelty, workload, study design, and writing. We will further correct errors and clarify all the concerns of the reviewers in the final version. We respond to the major points raised by the reviewers as follows.
Q1. The clinical relevance of generated mammogram in pCR prediction A1. Despite the advancements of MRI in breast imaging, generated mammograms continue to play a crucial role in therapy response prediction. A multimodal approach, incorporating both mammogram and MRI, provides the most comprehensive evaluation, enhancing the precision and reliability of breast cancer management.
Q2. Pathological information features extraction A2. This process is achieved through the application of a linear layer and the Exponential Linear Unit (ELU) activation function, to allow for the integration of pathological features.
Q3. Multi-view (CC/MLO) mammograms in pCR prediction experiments A3. For each patient, mammograms from both the CC and MLO views are used. The extracted features from the CC and MLO views are combined to form a comprehensive representation for each patient for therapy response prediction.
Meta-Review
Meta-review not available, early accepted paper.