List of Papers Browse by Subject Areas Author List
Abstract
In accelerated magnetic resonance imaging (MRI) reconstruction, the anatomy of a patient is recovered from a set of under-sampled measurements. Currently, unrolled hybrid architectures, incorporating both the beneficial bias of convolutions with the power of Transformers have been proven to be successful in solving this ill-posed inverse problem. The multi-scale strategy of the intra-cascades and that of the inter-cascades are used to decrease the high compute cost of Transformers and to rectify the spectral bias of Transformers, respectively. In this work, we proposed a dynamic Hybrid Unrolled Multi-Scale Network (dHUMUS-Net) by incorporating the two multi-scale strategies. A novel Optimal Scale Estimation Network is presented to dynamically create or choose the multi-scale Transformer-based modules in all cascades of dHUMUS-Net. Our dHUMUS-Net achieves significant improvements over the state-of-the-art methods on the publicly available fastMRI dataset.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1197_paper.pdf
SharedIt Link: https://rdcu.be/dV5C6
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72104-5_26
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1197_supp.pdf
Link to the Code Repository
N/A
Link to the Dataset(s)
BibTex
@InProceedings{Li_Dynamic_MICCAI2024,
author = { Li, Xiao-Xin and Zhu, Fang-Zheng and Yang, Junwei and Chen, Yong and Shen, Dinggang},
title = { { Dynamic Hybrid Unrolled Multi-Scale Network for Accelerated MRI Reconstruction } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15007},
month = {October},
page = {264 -- 274}
}
Reviews
Review #1
- Please describe the contribution of the paper
This study introduces a new deep learning architecture, termed Dynamic Hybrid Unrolled Multi-Scale Network (dHUMUS-Net), designed for 2D accelerated MRI reconstruction. The architecture integrates unrolled optimization techniques with a hybrid (transformer and convolutional-based) and multi-scale approach, combining U-shaped and pyramid-shaped multi-scale configurations. A key innovation of dHUMUS-Net is its dynamic capability to determine the most appropriate scale for reconstruction based on the input data. Both comparative and ablation studies demonstrate that this model surpasses several baselines and variations of the method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The manuscript is clear and well-structured, making the novel aspects readily comprehensible.
- The concept of dynamically predicting the maximum downsampling scale for a deep learning-based MRI reconstruction model is novel.
- Experimental results confirm that the proposed model outperforms the selected baseline models.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The dataset used includes multi-coil acquisitions from fastMRI, requiring the estimation of coil sensitivity maps. The manuscript does not address how these maps are estimated or refined.
- There is ambiguity regarding whether the MUST{s=1-4} modules in Equation 9 are trained independently from the main model, or if they are identical to those used in dMUST.
- The methodology implies a proportionality between the maximum scale and the acceleration factor, limiting the model’s functionality for acceleration factors greater than 8. Moreover, the method of constructing labels for the OSEN module is heavily dependent on the acceleration factor, raising questions about the necessity and effectiveness of the dynamic scaling.
- Related to the previous comment, the paper lacks an illustration or a discussion on the distribution of labels concerning different acceleration factors, for example, whether AR=8 corresponds to labels s=8. If these corelate, then is there really a need to dynamically quantify the scale?
- There is no analysis of computational times during inference, which is critical given the complex components (transformer-based architecture, iterative method, estimation of scale labels, etc) involved in the model, potentially impacting the practical utility of accelerating MRI procedures. It’s always nice seeing sophisticated architectures but let’s not forget that the goal is to accelerate MRI.
- Comparisons are limited to lesser-known methods and do not include several state-of-the-art methods cited within the paper (e.g. [5, 11, 22, 25, 26,….]), weakening the evaluation of the proposed model’s performance. Authors compare only to a not very established iterative method dated in 2019, namely PC-RNN, and two transformer-based architectures.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- The authors did not adhere to the guidelines for supplementary material, which restricted content to two pages of non-textual elements. While this will not influence my final recommendation, I will disregard the supplementary files.
Some minor comments:
- Assertions regarding the receptive fields at the end of page 1 lack adequate literary support and should be substantiated with additional references.
- Typo in the second paragraph of page 2: “import” -> “important”
- Clarification is needed for the term “high compressibility” mentioned in the last paragraph of Section 2.
- The paper refers to an illustration in Section 2 for explaining undersampling, which does not exist in the document.
- Typically, plots of training loss are less common; it would be more beneficial to focus on inference metrics.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
While the introduction of dHUMUS-Net represents a new deep-learning-based approach in MRI reconstruction, several critical aspects require clarification and improvement. Addressing the absence of details on coil sensitivity map estimation and the limited comparative analysis with state-of-the-art methods would make a stronger submission. Furthermore, considerations of computational efficiency should also be discussed.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Reject — should be rejected, independent of rebuttal (2)
- [Post rebuttal] Please justify your decision
Decision is due to weaknesses discussed above and mainly lack of clear comparison to sota methods not only performance-wise but also computationally-wise.
Review #2
- Please describe the contribution of the paper
The paper proposes a new architecture for MRI reconstruction namely Dynamic Hybrid Unrolled Multi-Scale Network (dHUMUS-Net). The authors combined the multi-scale strategy of two prior architecture, HUMUS-Net and ReconFormer. The intra-cascade architecture of HUMUS-Net and inter-cascade architecture of ReconFormer or PC-RNN have been combined in the proposed architecture. The experimental results shows that the proposed dHUMUS-Net outperforms one CNN-based and two transformer-based methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The method and architecture are well defined.
- The proposed designed architecture is a clever choice as it specifically targets spectral bias of transformers. Rectifying this bias can help in improving the generalization of the network across various types of MRI scans and anomalies.
- Equations in the methodology is well written.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- In the abstract it is written that “The multi-scale strategy of the intra-cascades and that of the inter- cascades are used to decrease the high compute cost of Transformers”. But in the paper, it is never mentioned how the proposed method decreases the high computation cost.
- Is the model suitable for other kinds of undersampling strategies like random sampling or VD undersampling? The authors can add one or two sentences based on their experiments about the performance of the model in different undersampling masks.
- One additional metric (i.e., PSNR) besides SSIM could have been used for comparison.
Minor corrections:
- In the table caption in section 4.3, it can be written what evaluation metrics has been used here. Although it has been defined earlier, but for more clarity it should be added in the caption.
- In section 4.1 there is an incomplete sentence. The last sentence of the first paragraph “In this work, we only the multi-coil datasets” is incomplete.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
No.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Based on the weaknesses, try to improve the paper.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well written as the proposed method and architecture of the model is well described. The authors also provided supplementary document. But it needs, some minor revisions to be accepted.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
Having reviewed the revisions and the detailed rebuttal provided, I find that all the concerns raised in my initial review have been fully addressed.
Review #3
- Please describe the contribution of the paper
This paper proposed a dynamic hybrid unrolled multi-scale network to reconstruct undersampled MRI. The network utilizes CNN to extract features from high-dimensional space and transformers for low-dimensional space. The transformer network is designed in a U-shape and the authors proposed an optimal scale estimation module to determine the optimal number of levels in U-shape. The network achieved better results than the comparison methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors improved a previously published network to determine an optimal number of levels in the transformer U-Net architecture. This change of network structure yielded better performance than the previous network and the authors also did an ablation study to investigate the effect of the number of U-net levels.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
In figure 2, it seems that the performance of max-scale 4 outperforms max-scale 8. Why is that? How does that influence the choice of range of max-scales? If the acceleration rate is higher than 8, do the authors recommend choosing a max range that’s the same as the AR? The authors only showed the SSIM evaluation results. It would be interesting to see other metrics such as NMSE and PSNR.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Do you have any additional comments regarding the paper’s reproducibility?
Public dataset is used and the network is not open-source.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Can the authors comment on the computational complexity of the network and time for training/testing?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents an improvement to an existing network by adding an optimal scale estimation network. The results seem to outperform the previous network.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
The authors addressed my concerns in the rebuttal. The authors may want to improve their writing to better deliver the RL concept if they feel it’s important.
Author Feedback
We have read the review comments carefully. After summary and combination, we got 8 major comments. Most of them, i.e., Comments 1~6, focus on the experiments or related details, where the main issues are caused by limited space. Comments 7~8 are for our method. The repetition level (RL) of the input data is the main factor of our method, but was ignored by reviewers. Our response is as follows.
- Reviewers #1 and #4 suggested reporting more evaluation metrics. Due to limited space, we only report SSIM results. Compared to NMSE/PSNR, SSIM is more important as radiologists care more about structure recovery. For completeness, we can put NMSE/PSNR results in revision.
- Reviewer #3 concerned the coil sensitivity map (CSM) estimation. We adopt the CSM estimation method used in E2E-VarNet, as our model uses CSM in operator A and its inverse, which is identical to Reduce and Expand operators in E2E-VarNet. We’ll illustrate this in revision.
- Reviewer #3 asked if the MUST modules used in Eq. 9 are trained independently from the main model. Yes, the MUST modules stacked in the main model are retrained. We’ll clarify this in revision.
- All reviewers concerned the computational cost of the proposed dHUMUS-Net. dHUMUS-Net decreases the compute cost of HUMUS-Net by using OSEN to tailor the MUST module in each cascade. As OSEN is a very small RNN, its compute cost can be neglected compared to the tailored time-consuming layers in the MUST module. As the time-saving effects are obvious, we didn’t give the time comparison results due to limited space. In practice, our method can run well on a machine with 11 GB GPU memory, while HUMUS-Net requires ≥16 GB GPU memory. We’ll discuss this in revision.
- Reviewer #3 indicated that plots of training loss are less common. We plotted the training loss to show how the training strategy affects the training procedure so that the readers can know more about the training details.
- Reviewer #3 indicated the compared methods are lesser known and should include more SOTA methods. The compared methods, PC-RNN, ReconFormer, and HUMUS-Net, were published very recently, which might make them lesser known to Reviewer #3. Reviewer #3 thought PC-RNN is not very established. However, PC-RNN is the champion model of the 2019 fastMRI Challenge and was further improved in [TMI-2022]. We can believe that PC-RNN is well established. We did not compare with the SOTA methods mentioned by Reviewer #3 because most of them are not very recent, not using multi-scales, and not SOTA till now. Due to limited space, we can only compare the very recent and relevant methods.
- Reviewer #3 considered the max-scale is proportional to the acceleration rate (AR), suggested discussing the distribution of labels against ARs, doubted if AR=8 corresponds to a fixed label, and if dynamic scaling is necessary. We believe that there is a misunderstanding. Our method is based on the fact that the optimal max-scale depends on the RL rather than on the AR. Although higher ARs tend to cause higher RLs and thus higher optimal scales, for a given AR, different MRI images containing different information might have very different RLs and thus different scales. Also, our method adopts a cascade structure, meaning different cascades taking different inputs with various RLs require different scales, inspiring the multi-scale strategy between cascades. Fig. 2 also showed the necessity of dynamic scaling.
- Reviewer #1 asked why max-scale 4 outperforms max-scale 8 in Fig. 2, the factors influencing the choice of range of max-scales, and how to choose the max-scale range if AR>8. As mentioned above, RL mainly affects the optimal max-scale. The better performance of max-scale 4 in Fig. 2 indicates that a max-scale of 8 is too high to fit the RL of the dataset. This result cannot be known in advance, which is why dynamic scaling is important. For ARs greater than 8, the choice of the max-scale range also depends on the RL of a given dataset.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
Reviews remain contardicting after the rebuttal with one reviewer reduced the overall score and the another increased. Overall the majority of the reviewers were positive. Main concern is the comparison with SOTA methods which while addressed by the author, one of the reviewers did not accept their argument. Yet, an interesting approach with publically available code.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Reviews remain contardicting after the rebuttal with one reviewer reduced the overall score and the another increased. Overall the majority of the reviewers were positive. Main concern is the comparison with SOTA methods which while addressed by the author, one of the reviewers did not accept their argument. Yet, an interesting approach with publically available code.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
Reviewer #3 holds concerns regarding the comparison with SOTA methods, but I believe the authors have successfully clarified their choice of the latest SOTA methods for comparison. Therefore, I would like to recommend acceptance.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Reviewer #3 holds concerns regarding the comparison with SOTA methods, but I believe the authors have successfully clarified their choice of the latest SOTA methods for comparison. Therefore, I would like to recommend acceptance.