List of Papers Browse by Subject Areas Author List
Abstract
Positron emission tomography (PET) is a well-established functional imaging technique for diagnosing brain disorders. However, PET’s high costs and radiation exposure limit its widespread use. In contrast, magnetic resonance imaging (MRI) does not have these limitations. Although it also captures neurodegenerative changes, MRI is a less sensitive diagnostic tool than PET. To close this gap, we aim to generate synthetic PET from MRI. Herewith, we introduce PASTA, a novel pathology-aware image translation framework based on conditional diffusion models. Compared to the state-of-the-art methods, PASTA excels in preserving both structural and pathological details in the target modality, which is achieved through its highly interactive dual-arm architecture and multi-modal condition integration. A cycle exchange consistency and volumetric generation strategy elevate PASTA’s capability to produce high-quality 3D PET scans. Our qualitative and quantitative results confirm that the synthesized PET scans from PASTA not only reach the best quantitative scores but also preserve the pathology correctly. For Alzheimer’s classification, the performance of synthesized scans improves over MRI by 4%, almost reaching the performance of actual PET. Code is available at https://github.com/ai-med/PASTA.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0501_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0501_supp.pdf
Link to the Code Repository
https://github.com/ai-med/PASTA
Link to the Dataset(s)
BibTex
@InProceedings{Li_PASTA_MICCAI2024,
author = { Li, Yitong and Yakushev, Igor and Hedderich, Dennis M. and Wachinger, Christian},
title = { { PASTA: Pathology-Aware MRI to PET CroSs-modal TrAnslation with Diffusion Models } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15007},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a conditional diffusion model based approach to translate MRI images to PET images for AD diagnosis. Several methods are proposed to preserve structural and pathological details after translation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Fusing multi-modality conditions is appreciated.
- The analysis for results is well done.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The descriptions for “Cycle Exchange Consistency” are very confusing: the conditioning network is trained via the regression loss in Eq. (1), which takes MRI images as inputs and produces MRI representations for DDPM. How is it then “reused for denoising”? Also, this part is missing / not clearly shown in Fig. 2 that describes the overall workflow.
- In Fig.2, notations such as h, t, c, are not explained.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- The “Cycle Exchange Consistency” part seems problematic to me as stated above. Although I get the central idea, I’m not quite sure how it is achieved by the described framework.
- Names of the methods could be revised, e.g., “Cycle Exchange Consistency”, “Conditioner arm”, etc.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Some parts of the method (stated above) require more clarifications.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes a PASTA I2I conditional diffusion model network, which utilizes a dual-arm structure capable of sharing high-dimensional features and pathological ROI priors of MetaROIs to enhance the network’s attention to pathological information during modal transformation. Combined with methods such as consistency loss, it further improves the consistency between the I2I output and input, as well as the 3D structure. The model compares different types of medical I2I networks and achieves optimal performance on the Alzheimer’s disease neuroimaging initiative (ADNI) database.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
PASTA is a novel I2I framework based on conditional diffusion models, aiming to generate synthetic PET images from MRI while preserving the structural and pathological details in the target modality. The innovation of this paper lies in the use of MetaROIs as pathological priors, enhancing the network’s sensitivity to pathological information. In the supplementary, the authors further supplemented experiments on Neurostat2 3D-SSP, verifying that the PET images generated by PASTA have a higher restoration of pathological information compared to other comparison networks. The work in this paper is comprehensive, with rich experiments conducted to support the advantages of the PASTA network.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I believe that there are still some shortcomings in this article, as follows: (1) Although the work in this paper is relatively comprehensive, validating the network on more medical image datasets could demonstrate its stronger generalization ability. (2) This paper did not explore and summarize the potential limitations of the network. (3)How do the number of network parameters and inference speed compare with other networks? Further experiments can be conducted for comparison.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Here are some constructive suggestions to address the shortcomings mentioned: (1) Consider including additional medical image datasets from different modalities or diseases to validate the network’s performance. This will provide a more comprehensive evaluation of the network’s generalization capabilities across a wide range of scenarios.
(2) Discuss the limitations of the proposed network, such as its sensitivity to noise or artifacts in medical images, its ability to handle complex pathologies, or its computational efficiency. Provide insights into potential reasons for these limitations and suggest directions for future research to address them.
(3)Conduct experiments to measure the number of parameters and inference speed of the proposed network. Compare these metrics with other state-of-the-art medical image analysis networks to provide a quantitative evaluation of the trade-off between accuracy and computational efficiency.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper introduces a novel Image-to-Image (I2I) conditional diffusion model network named PASTA, which effectively enhances the focus on pathological information during the modality conversion process by utilizing a dual-branch structure and MetaROIs’ pathological ROI prior. Experimental results on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database demonstrate that the PASTA network achieves superior performance compared to other types of medical I2I networks, validating its effectiveness in practical applications.
While the paper validates the network on the ADNI database, testing on more medical imaging datasets, particularly those from different modalities or diseases, would provide a more comprehensive demonstration of the network’s generalization ability. The paper lacks in-depth exploration and summary of the network’s potential limitations, such as sensitivity to noise or artifacts in medical images, capability to handle complex pathological conditions, and compu
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper presents an approach for image translation from MRI to PET using diffusion models. A cycle consistency ensures that pathology-preserving property of the method while AdaGN is used as a conditioning mechanism to include the guidance from the source MRI image at different stages in the denoising U-Net. The efficacy of the method is shown qualitatively and quantitatively.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
All included modules of the proposed method are reasonably well explained and each contribution is proved to be beneficial in the ablation study. The method shows very good results and additionally performs well in the downstream task of Alzheimer’s classification. The method is described well and the supplementary seems to contain all required information details to reproduce the study.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
A potential weakness of the method is that it requires paired image data. Although CycleGAN (which introduced the cycle consistency loss that the method uses) seems to deliver the worst results, it is actually based on unpaired data, thus it could be that it might benefit even from having access to more data from either source and/or target domain. In practice, one usually has access to a lot of unpaired data and limited amount of paired data.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Figure 2 seems slightly overwhelming. For a more comprehensive approach, the usage of block diagrams that are then explained in more detail might be helpful.
In Figure 1, PASTA also shows regions in the generated PET images that seems to metabolically defer a bit from the ground truth. A discussion on the practical implications on this can help to clarify whether this could be an issue or even a concern in a practical setting.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well-written and clearly presents the approach. Only minor comments/questions were raised.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We thank all reviewers for your time and effort in reviewing our manuscript. We are grateful for your insightful comments and constructive feedback, which not only confirm the quality of our work, but also help us to refine our arguments. We are truly happy that all reviewers acknowledged the novelty and the thorough analysis in our paper. Below, we would like to address some open remarks.
Descriptions for “Cycle Exchange Consistency” (R3) To clarify R3’s confusion, our Cycle Exchange Consistency has a unique cycling process, where the roles of the conditioner and denoiser arms are swapped in the two stages of the cycle (MRI→PET and PET→MRI). The two networks x_θ and ϕ_ω have the same architecture (UNet) but they operate independently. In Fig.3’s left side,x_θ serves as a conditioner in the G_m stage, but as a denoiser in G_p. ϕ_ω functions oppositely. This cycling approach requires only two trainable networks, x_θ and ϕ_ω. We will also release our code on GitHub for these implementation details.
Notations in Fig. 2 (R3) R3 noted that the “notations such as h, t, c, are not explained” in Fig. 2. We apologise that due to the page limit we only explained them in Sec. 2, in which h is the feature maps from the denoiser arm, t is the timestep, c is the clinical data, and h_m is the task-specific representation from the conditioner arm.
Requirement of paired image data (R4) R4 concerns that “a potential weakness of the method is that it requires paired image data”. While we agree that unpaired data can increase the training dataset, PASTA is specifically designed for applications where paired data is essential. Our goal is to generate PET images from MRI scans tailored to individual patients. For this reason, accurate and reliable translations are crucial, necessitating paired data to learn precise mappings between the modalities.
We also thank R1’s constructive suggestions on including discussions of limitations and experiments on additional datasets and computational efficiency, and R4’s comment on Fig. 2. Unfortunately due to the page limit we couldn’t elaborate too much on these aspects, but with brief illustrations in the conclusion and supplementary materials. We will try our best to cover them in our final version and release our code on GitHub for implementation details. We once again express our gratitude to all reviewers and eagerly await the application of our proposed methodology.
Meta-Review
Meta-review not available, early accepted paper.