Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Despite the high diagnostic accuracy of Magnetic Resonance Imaging (MRI), using MRI as a Point-of-Care (POC) disease identification tool poses significant accessibility challenges due to the use of high magnetic field strength and lengthy acquisition times. We ask a simple question: Can we dynamically optimise acquired samples, at the patient level, according to an (automated) downstream decision task, while discounting image reconstruction? We propose an ML-based framework that learns an active sampling strategy, via reinforcement learning, at a patient-level to directly infer disease from undersampled k-space. We validate our approach by inferring Meniscus Tear in undersampled knee MRI data, where we achieve diagnostic performance comparable with ML-based diagnosis, using fully sampled k-space data. We analyse task-specific sampling policies, showcasing the adaptability of our active sampling approach. The introduced frugal sampling strategies have the potential to reduce high field strength requirements that in turn strengthen the viability of MRI-based POC disease identification and associated preliminary screening tools.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2052_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/2052_supp.pdf

Link to the Code Repository

https://github.com/vios-s/MRI_Active_Sampling

Link to the Dataset(s)

https://fastmri.med.nyu.edu/

BibTex

@InProceedings{Du_The_MICCAI2024,
        author = { Du, Yuning and Dharmakumar, Rohan and Tsaftaris, Sotirios A.},
        title = { { The MRI Scanner as a Diagnostic: Image-less Active Sampling } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15003},
        month = {October},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper introduces an innovative machine learning-based framework that dynamically optimizes MRI k-space sampling at the patient level, using reinforcement learning for direct disease inference. This approach aims to enhance the accessibility and efficiency of MRI as a point-of-care (POC) diagnostic tool by reducing the need for full k-space sampling and high magnetic field strength, thus potentially enabling MRI use in settings with limited resources.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper proposes a novel patient-level active sampling strategy using a reinforcement learning framework.
    2. It addresses a critical need in medical imaging by enabling faster MRI processes and reducing equipment requirements.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    1. The policy network’s effectiveness depends significantly on the pre-trained classifier, which may not be robust across varied datasets or different undersampling patterns.
    2. As undersampling is mainly used to saving acquisition time and improve the applicability of MRI, however, this patient-level sampling require to at least double the time. If this is not a concern, may need to evaluate the computational expense and acquisition time.
    3. In terms of model design and training, please justify what the difference is between patient-level and population-level, since currently, I think it did not have much adaptation work but only used in different ways.
    4. Limited comparative experiments. The paper did not compare the proposed method with other k-space sampling methods, including population-level methods mentioned in introduction.
    5. Lack of comparison of reconstructed images.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. Provide a more detailed comparative analysis with existing methods, particularly those that use similar machine learning approaches but differ in implementation details such as sampling strategies or learning algorithms.
    2. Investigate the feasibility and potential benefits of simultaneously training the classifier and the policy network, which might enhance the adaptability and accuracy of the sampling policy.
    3. Provide quantitative and qualitative comparisons of reconstruction results. The reconstructed MRI images from the output sampling policy should be illustrated, showing the effectiveness of your sampling strategy in maintaining the integrity and resolution of the images.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Reject — should be rejected, independent of rebuttal (2)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Firstly, the dependency of the policy network on a pre-trained classifier raises concerns about the robustness of the framework across diverse datasets and undersampling patterns. Additionally, the increased acquisition time associated with patient-level sampling is a notable drawback that needs careful consideration, especially regarding computational expense and practicality. Moreover, the lack of comparative experiments, particularly against population-level methods and the absence of reconstructed image comparisons.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Reject — should be rejected, independent of rebuttal (2)

  • [Post rebuttal] Please justify your decision

    I appreiciate the authors for the clarification. However, I believe the motivation of designing the patient-level mask is still unclear. The performance of the proposed model is marginally improved compared to the policy reconstruction network. The comparison experiments are not sufficient to validate the model performance.



Review #2

  • Please describe the contribution of the paper

    The paper proposes an active sampling reinforcement learning method to directly classify meniscus tear from undersampled k-space data. The proposed method achieves comparable, performance to [2], which has a two step process that reconstructs the image in a first step.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Paper is well-written and easy to follow.
    • The authors provided a preliminary version of the code.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • Lacks detailed motivation, or results, as to why not acquiring or using the MRI is important or useful in practice i.e. classifying meniscus tear directly from the k-space data is useful. This is the difference between the paper and [2,11,20]. This point is very important, as the proposed approach underperforms the baseline in figure 3.
    • Only uses a single baseline [2]. The paper should also consider using [11,20] as baselines.
    • No statistical analysis on the results.
    • In figure 3, the policy reconstruction baseline (reconstructing the image with [2], then using deep learning classifier), outperforms the oracle (deep learning classifier on the ground truth image). I suspect this is because the classifier network hyperparameters (the ResNet-50) were not optimized on the oracle data (recommended).
    • The paper downsamples the data, which is not done by e.g. [2].
    • The paper only considers a single task on the data (classifying meniscus tear), why did you choose this task and not others? See [21]-table-1 for alternative options.

    I look forward to the authors’ rebuttal.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?
    • Data is all open-source.
    • I am pleased the authors released a (preliminary) version of the code. Feedback on the code is below.
  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    • Avoid using passive tense when writing.
    • Add ‘proposed’ to legend in figure 3, also make the legend bigger.
    • Advice on the code:
      • The README.md file only has a title. It is difficult to understand what to run, what the data is et.c.
      • See https://github.com/paperswithcode/releasing-research-code for example release code requirements.
      • use a requirements.txt file using pip to export the dependencies, which prevents the very complicated conda dependencies from conda export
      • be careful with anonymity in environment.yml line 311.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please see weaknesses section.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    Thank you for the rebuttal. I am still not entirely convinced that avoiding the image reconstruction is clinically useful (even after reading [14]) - in any case the authors followed a paradigm presented at a MICCAI conference, which is legitimate. The rebuttal is good and I will raise to weak accept.



Review #3

  • Please describe the contribution of the paper

    In this paper, an approach for designing an MRI under-sampling procedure based on reinforcement learning is presented, which does not rely on intermediate image reconstruction, but acts directly on under-sampled k-space to optimize a disease classification objective. This approach is evaluated in the public fastMRI and fastMRI+ datasets, showing performance approaching that of reconstruction-based methods.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    One strength of the paper is the patient-specific undersampling masks that can be learned, as opposed to learning population level masks that are fixed for each patient. Another strength is the fact that no image reconstruction needs to be performed, therefore there are no dependencies on an appropriate reconstruction network tuned for the data distributions in question.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    One weakness of the paper is that its performance is still below that of the Policy Reconstruction approach. Another weakness is the claim that no fully-sampled data is needed in the proposed approach for policy training. However, this does not seem possible, because for training, to train the sampler S, all possible k-space lines must be available to sample, therefore the data used for training must be fully sampled, so it would seem that it has the same data training requirements as the Policy Reconstruction approach.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    A clear high level description of the method is provided, and a link to code is provided.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The paper could present a better case for the utility of the proposed approach in light of the fact that the performance lags the reconstruction-based method. One way to do this would be to demonstrate explicitly training of the Active Sampler without any fully-sampled training data, as claimed possible in Section 5.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents an interesting idea, based off prior work in RL for MRI sampling mask selection, but removes the need for image reconstruction. However, the results do not out-perform reconstruction based approaches, and the utility of the approach is somewhat in question.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We thank all reviewers for their positive and constructive feedback. We now clarify a few points raised.

Q.Motivation of direct inference (meniscus tear) from k-space (R3,R4) As mentioned in the introduction, [14] shows biomarkers can be obtained directly from k-space. Full image reconstruction thus shouldn’t be necessary, allowing for k-space savings that can lead to quicker acquisition or reduced field strength. Thus, we aim to directly optimize k-space acquisition for inference without reconstruction. Evaluating image reconstruction results are in scope but will change the inference mechanism. Using meniscus tear detection as the exemplar task demonstrates the applicability of a portable MRI device for sports events [10]. Subsequent experiments showed that we can also identify cartilage thickness loss as another task.

Q.Difference between patient and population-level mask (R4) A population-level mask does not use any policy during acquisition but a prefixed mask. Our approach samples the mask one line at a time. Fig 5 does show masks between patients vary. Thus, a population-level mask would either over-acquire k-space lines for some patients or under-acquire for others. To improve clarity, we will include a population level mask for comparison.

Q.No fully-sampled data are needed for policy training (R1) We believe the reviewer was confused. We claim in Secs 4.3 and 5 that our method does not require ‘high-fidelity’ fully sampled k-space data, which is necessary for Policy Recon. During training, the policy network may sample any possible line from the action space, which importantly can be low-quality k-space e.g. collected from low-field Portable MRI. Notably, the training of the policy and classifier happen offline. During inference, we do not assume existence of all k-space lines for the policy: it is the policy that dictates the sampling of the next line.

Q.How our method performs vs. using reconstruction quality as reward? (R1,R3) Results in Fig 3 show that we outperform a reconstruction policy on specificity while matching its AUC and recall, providing competitive results. A key reason is the (un)certainty of the reward. As noted in Sec 4.3, p.4, our policy uses the noisy reward of the classifier, whereas the reconstruction policy uses a high-fidelity image and its label. Differences between Policy Recon and Oracle could be due to hyperparameter choices, but more likely, see footnote of p. 6, are the effects of denoising and smoothing on the reconstruction, which act as a regularizer for the Oracle classifier.

Q.Computational cost of patient-level (R4) Obtaining a patient-level mask does incur a computation cost vs a random undersampling mask, but this is not double and does provide better performance with identical acquisition time. In fact, optimizing the mask shows better results at the same sampling rates compared to random undersampling mask in Fig 3. [2,11] also evidence in image reconstruction, patient-level masks achieve better performance for the same sampling rate vs a random mask.

Q.Why choose [2] as a baseline? (R3) We agree [11,20] can serve as baselines, but we opted on [2] for a fairer comparison. Our method uses the same greedy policy sampler backbone as [2], allowing a direct comparison of k-space acquisition effects between reconstruction and our direct inference. As noted in Sec 5.1, p. 6, [2] also downsamples the dataset for computational expedience. Including [11,20] is possible, but their different policy network designs deviate from our main hypothesis: saving lines by avoiding reconstruction, thus out of scope for this work.

Q.Two stage vs. simultaneously training? (R4) Indeed, policy performance depends on the pre-trained classifier as it ensures a stable environment. However, any bias or shift in the classifier will propagate to the policy decisions. We are currently investigating joint training of these components (e.g., by fine-tuning the classifier) to address domain shifts.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The paper presents an interesting concept, though its results do not outperform existing reconstruction-based approaches. Nonetheless, it addresses a critical need in medical imaging by enabling faster MRI processes and reducing equipment requirements. The authors have effectively addressed the reviewers’ concerns, and I am happy to recommend accept.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The paper presents an interesting concept, though its results do not outperform existing reconstruction-based approaches. Nonetheless, it addresses a critical need in medical imaging by enabling faster MRI processes and reducing equipment requirements. The authors have effectively addressed the reviewers’ concerns, and I am happy to recommend accept.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    the authors did a good job with the rebuttal and the majority of reviewers is for accept now.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    the authors did a good job with the rebuttal and the majority of reviewers is for accept now.



back to top