List of Papers Browse by Subject Areas Author List
Abstract
Magnetic Resonance Imaging (MRI) is a widely used noninvasive medical imaging technique that provides excellent contrast for soft tissues, making it invaluable for diagnosis and intervention. Acquiring multiple contrast images is often desirable for comprehensive evaluation and precise disease diagnosis. However, due to technical limitations, patient-related issues, and medical conditions, obtaining all desired MRI contrasts is not always feasible. Cross-contrast MRI synthesis can potentially address this challenge by generating target contrasts based on existing source contrasts. In this work, we propose Contrast Representation Learning (CRL), which explores the changes in MRI contrast by modifying MR sequences. Unlike generative models that treat image generation as an end-to-end cross-domain mapping, CRL aims to uncover the complex relationships between contrasts by embracing the interplay of imaging parameters within this space. By doing so, CRL enhances the fidelity and realism of synthesized MR images, providing a more accurate representation of intricate details. Experimental results on the Fast Spin Echo (FSE) sequence demonstrate the promising performance and generalization capability of CRL, even with limited training data. Moreover, CRL introduces a perspective of considering imaging parameters as implicit coordinates, shedding light on the underlying structure governing contrast variation in MR images. Our code is available at
https://github.com/xionghonglin/CRL_MICCAI_2024.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1236_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: N/A
Link to the Code Repository
https://github.com/xionghonglin/CRL_MICCAI_2024
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Xio_Contrast_MICCAI2024,
author = { Xiong, Honglin and Fang, Yu and Sun, Kaicong and Wang, Yulin and Zong, Xiaopeng and Zhang, Weijun and Wang, Qian},
title = { { Contrast Representation Learning from Imaging Parameters for Magnetic Resonance Image Synthesis } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15007},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose a method of harmonization based on the acquisition sequence parameters. The method trains an encoder and decoder to synthesize different image contrasts. It uses cross-attention with the input and output sequence parameters to take encoded representations in and out of a common space.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The method uses contrast representation learning for this task, which is a good choice for the topic. This model has good potential, and the initial results presented here demonstrate feasibility.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The experiments here are very preliminary, only looking at 10 subjects, and restricted to TSE-based acquisitions. The low sample size and lack of a testing/validation set mean that the network has seen each subject many times before (only 25 slices per volume) and could likely memorize the structure of each brain. This is an assumption but should be disproven to demonstrate generalizability. Similarly, the lack of different imaging techniques increases the importance of the selected acquisition parameters. In TSE imaging, nearly all contrast is determined by these three parameters. However, different sequences are more complex and might not fit so nicely in this model. There is also no effect mentioned of turbo factor (echo train length). A longer echo train leads to a longer effective echo time, which can alter the contrast of an image (especially T2-weighted images).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Comments/Questions: • Why was TI determined empirically? This should have been available from the scanner. • The withheld contrasts were both T2-weighted, and one was very similar to another T2-weighted image. It would have been more impactful to withhold an entire contrast (T1 or T2 FLAIR). This would have demonstrated more applicability. • A withheld set of subjects must be used for testing. This small data pool makes that difficult, but any results without this call into question generalizability. This method seems to be exploring some avenue of unsupervised training, but this cannot determined from these small experiments. • There is a typo in Figure 4. #5 should be T2-weighted. #2 is PD-weighted.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper has well-designed experiments and a good theoretical basis, but the experiments are limited. A small sample size and potentially no withheld testing subjects mean that the process has the potential for not being generalizable. A greater variety of images and a larger sample would make this paper more impactful.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Reject — could be rejected, dependent on rebuttal (3)
- [Post rebuttal] Please justify your decision
The authors addressed many concerns of the reviewers as much as possible within the rebuttal rules. However, this is still an limited preliminary analysis of this method. There are a number of public datasets which disclose their parameters and could be used for training/testing. I respect the use of prospectively acquired data for consistency, but the kinds of acquisitions (TSE only and mainly T2-weighted) and limited test set (only 2 subjects with no cross validation) severely limit the current impact of this method. The weaknesses presented by multiple reviewers still stand.
Review #2
- Please describe the contribution of the paper
This paper introduces Contrast Representation Learning (CRL), a new method for synthesizing Magnetic Resonance Imaging (MRI) contrasts. CRL learns an contrast representation, and introduce the mechanism of modulating scanning parameters, which is indeed a smart way for a more generate purpose contrast synthesis.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper tackles an important problem of image synthesis, how to make it flexible and generalizable. The idea of modulating the scanning parameter through cross attention is smart and effective.
- The presentation is clear and very comprehensive!
- The author compared the method with other SOTA methods, both quantitatively and qualitatively.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
Lack of analysis of the architecture and dimensionalities, I would like to know how the dimensionalities of the contrast representation impact the results, this would give us an insight of how compressed the representation can be.
-
Some metrics other than PSNR and SSIM would be helpful, for example LPIPS, FID, or radiology evaluations.
-
Would appreciate some evaluations on how the model performs for pathologies, lets say when there is a tumor, that’s actually where we care the most.
-
One interesting analysis that is missing is how the number of training sequences impact the results. Intuitively, if we only have two contrasts, the results will be suboptimal, denser sequence space should help.
-
Missing references: 1) Wang, K., Doneva, M., Meineke, J., Amthor, T., Karasan, E., Tan, F., … & Lustig, M. (2023). High‐fidelity direct contrast synthesis from magnetic resonance fingerprinting. Magnetic Resonance in Medicine, 90(5), 2116-2129. 2) Qiu, S., Ma, S., Wang, L., Chen, Y., Fan, Z., Moser, F. G., … & Li, D. (2023). Direct synthesis of multi‐contrast brain MR images from MR multitasking spatial factors using deep learning. Magnetic Resonance in Medicine, 90(4), 1672-1681. and other recent works
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
See the weakness comments.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Very nice paper, would be happy to increase my score after the rebuttal.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper proposed a novel Contrast Representation Learning (CRL) method to perform MR image synthesis. The proposed method considers imaging parameters as implicit coordinates for generating new contrast, rather than performing end-to-end cross-domain mapping as previous generative models did. The proposed method shows better synthesis imaging quality than previous methods, and can generate contrast images that are not encountered during training.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The paper proposed a novel framework for the MR image synthesis. They consider imaging parameters as implicit coordinates in a high-dimensional manifold. The framework is novel, concise and useful.
-
The paper developed proper network structures to learn the relationship between contrast-related features and unrelated features, including using cross-attention to incorporate imaging parameters, and using structural component in SSIM to enforce structural consistency between features of different contrast images.
-
The proposed method shows better PSNR and SSIM than previous methods in generating synthesis MR images with different input-output combinations. And the proposed method only needs to be trained a single model for the different combinations.
-
The proposed method can generate new contrast images that were not used in training, which is promising for clinical applications.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Different imaging parameters may produce images with different structures even with the same subject. The structural consistency constraints in the proposed method may be not good in some input/output sequence combinations.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
-
There seems to be some typos in section 3.4. The authors firstly stated to use sequence 7 (T2) to exemplify the synthesis of all other sequences, but later that becomes “by using sequence 5 as input to synthesize images”. The caption of Fig. 4 also shows “using sequence 5 (T2) as input”. The authors should clarify them.
-
I would like to know how the structural consistency loss performs with different sequence combinations. And is it related to the differences of performances with different input-output combinations shown in Table 3?
-
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposed a novel framework for the MR image synthesis. The framework is novel, concise and useful.
The proposed method shows better imaging fidelity than previous methods in generating synthesis MR images with different input-output combinations. And the proposed method only needs to be trained a single model for the different combinations.
The proposed method can generate new contrast images that were not used in training, which is promising for clinical applications.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
I am satisfied with the rebuttal.
Author Feedback
R1: How the dimensionality of the latent contrast representation affects the performance. A: We experimented with different dimensionalities (64, 128, 256, 512) for the latent contrast representation. The model performance was suboptimal below 256, while the improvement was marginal above 256. Therefore, we chose 256 in this paper. We believe the representation should not be over-compressed as it must sufficiently represent brain structural information. The specific dimensionality also depends on the model architecture, which is not the focus of our paper. Our aim is to explore the latent relationship between contrast and imaging parameters using a multi-sequence dataset and imaging parameters as conditions for the network.
R1: More evaluation metrics. A: We agree that including more metrics would provide a more comprehensive evaluation. However, due to rebuttal rules, we cannot include new experimental results in this study. We are currently investigating generating images with novel contrast using our method for segmentation tasks, which would serve as a valuable evaluation of our model’s capabilities.
R1: Influence of the number of sequences on the results. A: Using more sequences in training leads to better results. We balanced scanning time while trying to cover the entire parameter space, selecting nine sequences. However, performance is not entirely proportional to the number of sequences, as certain sequences are crucial for training (e.g., sequence 8: TR 8000ms, TE 99.98ms, TI 1030ms). Without this sequence, the model becomes difficult to converge. Investigating sequence selection and underlying mechanisms is an important future research direction.
R1: Evaluation of pathology data such as MRI with tumor. A: Evaluating the model’s performance on pathology data is important. Unfortunately, we lack such in-house data, and public datasets like BraTS do not disclose imaging parameters, making them unsuitable. We appreciate the comment as it aligns with our future research direction. The presence of tumors will make the task more challenging due to their non-fixed contrast compared to normal brain tissues. Dealing with tumor characteristics is a difficult yet worthwhile problem to solve.
R3: Lack of test set. A: We would like to clarify that we do have a test set. As mentioned in ‘Implementation Details’, we used eight subjects for training and two subjects exclusively for testing. These two subjects used for testing were never exposed to the network during training.
R3: Generalizability concerning small dataset. A: While the number of subjects may appear small, there are 90 scans used in this study (each subject underwent 9 scans). During training, the network performs mutual image translation between any two sequences, effectively enlarging the actual training data size.
R3: Experiments restricted to TSE-based acquisitions. A: We agree with validating our method on a wider range of techniques. Our study is a preliminary exploratory experiment, and we used relatively common TSE sequences. The success on TSE suggests potential for extension to broader MRI sequences.
R3: Lack of effect of echo train length (ETL). A: In our acquisition, ETL significantly impacts scanning time, so we didn’t acquire multiple sequences by varying ETL while fixing other parameters. This is a very constructive suggestion, and we will include ETL as a variable parameter in future dataset expansions.
R4: Structural consistency constraint. A: We acknowledge potential structural differences across sequences for the same subject. To address this, we modified the SSIM loss function, focusing on a component that relates to image structure. This component computes the covariance of two images divided by the product of their variances. By emphasizing this structural component, we aimed to make the consistency constraint less strict while still encouraging structural similarity across sequences.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
Three reviewers consistently acknowledge the novelty of this work in brain MR image synthesis. There is a concern from R3 about the need of more dataset for evaluation. But this might not be feasible for new applications which start from small datasets.
I vote for accept considering its novelty in concepts and techniques as acknowledged by all reviewers.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Three reviewers consistently acknowledge the novelty of this work in brain MR image synthesis. There is a concern from R3 about the need of more dataset for evaluation. But this might not be feasible for new applications which start from small datasets.
I vote for accept considering its novelty in concepts and techniques as acknowledged by all reviewers.