Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

To accelerate Magnetic Resonance (MR) imaging procedures, Multi-Contrast MR Reconstruction (MCMR) has become a prevalent trend that utilizes an easily obtainable modality as an auxiliary to support high-quality reconstruction of the target modality with under-sampled k-space measurements. The exploration of global dependency and complementary information across different modalities is essential for MCMR. However, existing methods either struggle to capture global dependency due to the limited receptive field or suffer from quadratic computational complexity. To tackle this dilemma, we propose a novel Frequency and Spatial Mutual Learning Network (FSMNet), which efficiently explores global dependencies across different modalities. Specifically, the features for each modality are extracted by the Frequency-Spatial Feature Extraction (FSFE) module, featuring a frequency branch and a spatial branch. Benefiting from the global property of the Fourier transform, the frequency branch can efficiently capture global dependency with an image-size receptive field, while the spatial branch can extract local features. To exploit complementary information from the auxiliary modality, we propose a Cross-Modal Selective fusion (CMS-fusion) module that selectively incorporate the frequency and spatial features from the auxiliary modality to enhance the corresponding branch of the target modality. To further integrate the enhanced global features from the frequency branch and the enhanced local features from the spatial branch, we develop a Frequency-Spatial fusion (FS-fusion) module, resulting in a comprehensive feature representation for the target modality. Extensive experiments on the BraTS and fastMRI datasets demonstrate that the proposed FSMNet achieves state-of-the-art performance for the MCMR task with different acceleration factors.



Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2570_paper.pdf

SharedIt Link: https://rdcu.be/dV5BA

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72104-5_6

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/2570_supp.pdf

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Che_Accelerated_MICCAI2024,
        author = { Chen, Qi and Xing, Xiaohan and Chen, Zhen and Xiong, Zhiwei},
        title = { { Accelerated Multi-Contrast MRI Reconstruction via Frequency and Spatial Mutual Learning } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15007},
        month = {October},
        page = {56 -- 66}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a new multi-contrast MR reconstruction network to accelerate the MR imaging process by using fast imaging modality data to assist slow imaging modality data. Similar ideas have been explored before, but there are some novel ideas in the network architecture.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This paper introduces a novel frequency-space mutual learning network to efficiently explore global dependency information across different modalities. By constructing FSFE, CMS-fusion, and FS-fusion modules, effective utilization of auxiliary modality information is achieved. Extensive experiments validate the effectiveness of the proposed method.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1、There are already some methods that utilize fast imaging modalities to assist slow imaging modalities, such as [5] and [8]. What are the advantages of your method? 2、The authors mentioned the issue of quadratic computational complexity in the abstract but did not discuss corresponding solutions in the subsequent text. 3、In the frequency branch of the FSFE module, would it be better to use complex convolution directly instead of separating into magnitude and phase components? 4、In Equation 7, the operations described by A(·) and P(·) act on the Fourier domain, but the variable I in the equation appears to be in the image domain, requiring further explanation. 5、Splitting the dataset into training, validation, and test sets in a ratio of 7:1:2 or 8:1:1 seems more reasonable. 6、From the quantitative metrics in Table 1, the overall results on the fastMRI dataset appear to be relatively low. Supplementing with qualitative results may provide a better understanding.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    I hope the authors can provide open-source code.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    1、The quantitative results section could include the results of statistical analyses, such as t-tests, to validate the effectiveness of the method.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The idea of using fast imaging modalities to assist slow imaging modalities has already been applied, with no novelty in its application. The network architecture has some innovations but is not sufficient to meet the requirements. Additionally, the quantitative results on fastMRI are low, requiring visualized results to verify the model’s performance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    Thank you for the author’s response, which has mostly clarified my concerns.



Review #2

  • Please describe the contribution of the paper
    1. The Fourier transforms are used to capture global dependency with an image-size receptive field.
    2. Global features from the frequency branch and local features from the spatial branch are selectively incorporated and fused to fully use the complementary information from the auxiliary modality.
    3. The proposed FSMNet outperforms existing multi-contrast MR reconstruction methods across various acceleration factors.
  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. It is novel and interesting to use the global property of Fourier transforms to capture global dependency with an imagesize receptive field. Previously, researchers mainly enlarge receptive fields by deepening networks or using Transformers. The authors proposed a new viewpoint to capture global features.
    2. Attention mechanisms are well used in the CMS-fusion module and FS-fusion module of the proposed FSMNet. In particular, Transformers are very well used in the FS-fusion module, see Eq. (5), where the enhanced global features and local features are used as the Q and K/V, respectively.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The authors used Fourier transform to capture global features. It’s really a novel idea. However, the underlying principles and the experimental illustrations that can demonstrate the global effects caused by FFT are not well shown. We can only see the following descrition. “According to Fourier transform theory [13], each pixel in the Fourier domain interacts with all pixels in the spatial domain, enabling image-size receptive field and global feature extraction by extracting features in the Fourier domain.” I doubt this description due to two reasons. First, fully-connected network can also capture the global features. Why were fully-connected network rarely used to boost performance? Second, if FFT can really achieve the global effect, I think it should be a common principle and can also be used to reconstruct natrual images. However, FFT is rarely used in natural image reconstruction. So, does it mean that FFT has a close relation with the k-space characteristic of the MRI data? Moreover, some presentations are not clear enough.

    1. The authors indicated that the output of the frequency branch of FSFE can be obtained via inverse Fourier transform (IFFT). However, it is not clear how the amplitude and phase components can be used to perform IFFT.
    2. The function f(.) in Eq. (5) was not defined.
    3. How can we get two outputs from the Frequency-Spatial fusion (FS-fusion) module? According to the illustration of the FS-fusion module in Fig. 1 (c), only F_ts can be obtained. But why did the authors draw two output lines from FS-fusion module, as shown in the left part of Fig. 1? And how can we obtain the \(I_{tar}^{fre}\)in the final ouput of FSMNet?
    4. Why were only\(I_{tar}^{fre}\) used in frequency-level loss, i.e., Eq. (7)? Why not also use \(I_{tar}^{spa}\)?
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    See the weaknesses part.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The underlying principle of using FFT to capture the global features was not clearly described in the paper.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposed a novel network named FSMNet, which captures global dependencies across different modalities. FSFE module, which consists of a frequency branch and a spatial branch, extracts the feature for each modality. Then CMS-fusion module and FS-fusion modules incorporate the frequency and spatial features to generate a comprehensive feature for the MRI reconstruction.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This paper exhibits clear and well-structured content and achieves good reconstruction performance.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    There are some unclear points in the methods and experiment part.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Could you release the test code or demo to facilitate the reproducibility of proposed method before the rebuttal deadline?

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Introduction No comments

    Methodology No comments

    Experiments

    1. Could you show the sample mask under different acceleration factors?
    2. The right-top sub-image is not aligned with the top border.
    3. In table 2. we do observe the difference between networks with and without CMS-fusion, FS-fusion module. Is the difference due to the change of parameters amount when module is included or exclude? You should ensure the model under different configurations have the similar parameter amount.
    4. Results like Fig2 for fastMRI dataset should be displayed.

    References: 1. k-space deep learning for accelerated MRI

    Adaptive convolutional neural networks for accelerating magnetic resonance imaging via k-space data interpolation

    Multiple Slice k-space Deep Learning for Magnetic Resonance Imaging Reconstruction

    Its best to cite these 3 papers related to your work.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    .

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We sincerely thank all Reviewers for their constructive comments, especially the weak accept with very confident from R1 and R2. We will release the source code and model once the paper is accepted. Below please find the responses to the main concerns.

R1Q1: Unclear description of global effects caused by FFT? Why were FC networks rarely used to boost performance? Does FFT have a closer relationship with MRI data? 1)The Fourier transform decomposes an image into a sum of sinusoidal waves with various frequencies. Altering a single pixel in the Fourier domain affects the entire image once transformed back into the spatial domain, resulting in a global effect. Similarly, performing convolution operations in the Fourier domain influences the entire image, effectively providing an image-sized receptive field. 2)Although FC networks can capture global features due to their dense connections, they are limited by high computational costs and poor generalizability. The number of parameters in FC networks scales quadratically with the feature size, making them especially computationally expensive for high-resolution features. Furthermore, FC networks lack flexibility regarding feature size. In contrast, utilizing FFT offers a more computationally efficient and flexible alternative to capture global features. 3)The use of FFT to achieve global effects in natural image reconstruction is also a popular topic, as demonstrated by works like “Focal Frequency Loss for Image Reconstruction and Synthesis (ICCV’21)” and “Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder (CVPR’23)”. Furthermore, FFT has a closer relationship with MRI data, as MRI signals are typically acquired in k-space and then converted into images via inverse FFT. Therefore, using frequency signals in MRI data aligns with the inherent characteristics of the data.

R1Q2: how to perform IFFT with amplitude and phase components. We first compute the real and imaginary parts from the amplitude and phase components via real = ampcos(pha) and imag = ampsin(pha), then perform IFFT to obtain the image = torch.abs(torch.fft.irfft2(torch.complex(real, imag))).

R1Q3-Q5: Unclear presentations. The function f(.) in Eq. (5) represents several convolution layers. In the FS-fusion module, the output shown in Fig. 1(c) serves as the output of the spatial branch, while the output of the frequency branch remains unchanged from its input. We will clarify these details in the revised version.

R2Q1: Under-sampling masks. We follow the official implementations of the fastMRI dataset. For the 4x and 8x AF, the central 10% and 4% of k-space lines are fully sampled, respectively. The remaining lines are sampled randomly for 4x and equidistantly for 8x AF.

R2Q3: Fairness of ablation study. For the variants w/o the CMS-fusion or FS-fusion module, we added several convolution layers after feature summation to ensure a similar number of parameters in the ablation study.

R3Q1: Novelty and advantages of this work. We do not claim the introduction of MCMR as our contribution but rather propose a novel method to address the MCMR problem. Compared to existing studies in MCMR, we are the first to utilize the Fourier transform to efficiently capture global dependencies and propose the novel CMS-fusion and FS-fusion modules to integrate information across different modalities and domains.

R3Q2: Discussion of computational complexity. It is well known that the computational complexity of transformer layers is significantly higher than that of convolutional layers. We apply convolutional layers in the Fourier domain to extract global properties. The variant that replaces the frequency branch with transformer layers has 819.2M parameters and 675.1G FLOPs, compared to our method with 14.1M parameters and 147.6G FLOPs.

R3Q4&Q6: Explanation for Eq. (7) and qualitative results on the fastMRI dataset. More explanations and qualitative results will be provided.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



back to top