Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Although single-task medical image restoration (MedIR) has witnessed remarkable success, the limited generalizability of these methods poses a substantial obstacle to wider application. In this paper, we focus on the task of all-in-one medical image restoration, aiming to address multiple distinct MedIR tasks with a single universal model. Nonetheless, due to significant differences between different MedIR tasks, training a universal model often encounters task interference issues, where different tasks with shared parameters may conflict with each other in the gradient update direction. This task interference leads to deviation of the model update direction from the optimal path, thereby affecting the model’s performance. To tackle this issue, we propose a task-adaptive routing strategy, allowing conflicting tasks to select different network paths in spatial and channel dimensions, thereby mitigating task interference. Experimental results demonstrate that our proposed \textbf{A}ll-in-one \textbf{M}edical \textbf{I}mage \textbf{R}estoration (\textbf{AMIR}) network achieves state-of-the-art performance in three MedIR tasks: MRI super-resolution, CT denoising, and PET synthesis, both in single-task and all-in-one settings. The code and data will be available at \href{https://github.com/Yaziwel/All-In-One-Medical-Image-Restoration-via-Task-Adaptive-Routing.git}{https://github.com/Yaziwel/AMIR}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0787_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0787_supp.pdf

Link to the Code Repository

https://github.com/Yaziwel/All-In-One-Medical-Image-Restoration-via-Task-Adaptive-Routing.git

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Yan_AllInOne_MICCAI2024,
        author = { Yang, Zhiwen and Chen, Haowei and Qian, Ziniu and Yi, Yang and Zhang, Hui and Zhao, Dan and Wei, Bingzheng and Xu, Yan},
        title = { { All-In-One Medical Image Restoration via Task-Adaptive Routing } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15007},
        month = {October},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    In this paper, the authors propose a novel All-In-One Medical Image Restoration via task-adaptive routing strategy, allowing conflicting tasks to select different network paths in spatial and channel dimensions, thereby mitigating task interference.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The authors introduce an innovative All-in-one Medical Image Restoration(AMIR) network capable of handling multiple MedIR tasks with a single universal model. The innovation is based specifically on the incorporation of the task-adaptive routing which involves routing instruction learning, spatial routing, and channel routing. -The experimental results on several tasks and imaging modalities (MRI, CT and PET) seem to be reasonable and promising.
    • Extensive experiments show the proposed approach achieves state-of-the art performance in both single-task MedIR and all-in-one MedIR tasks.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    -There are some typos in the paper, such as « perrformance » -The number of samples used for validation is so limited which may affects the generalizability of the proposed MedIR approach, specifically with CT dataset. -it could be interesting that the authors study the statistical significance of the results when compared to the state-of-the-art approaches. -the computation complexity of the proposed approach when compared to the state-of-the art approach should be discussed.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    -There are some typos in the paper, such as « perrformance » -The number of samples used for validation is so limited which may affects the generalizability of the proposed MedIR approach, specifically with CT dataset. -it could be interesting that the authors study the statistical significance of the results when compared to the state-of-the-art approaches. -the computation complexity of the proposed approach when compared to the state-of-the art approach should be discussed.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The innovation of the proposed Medical Image Restoration approach.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    The authors explore an all-in-one medical image restoration tasks, motivated by the usual performance drop when single task models are applied to a different domain. To address this issue, they propose an all-in-one medical image restoration network integrating task adaptive routing. Their approach incorporates a Routing Instruction Network along with Spatial and Channel Routing Modules. The authors demonstrate the performance of their model across three tasks for both single and multi-task applications. In an ablation study they studied the effect on task combinations and the task adaptive routing strategy.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is meticulously structured, offering readers a clear and organized path through its content. It includes a detailed graphical representation of the network architecture, aiding in understanding the methodology effectively. Moreover, the empirical performance surpasses recent baselines, highlighting the effectiveness of the proposed approach. Furthermore, the commitment to making both the code and data available enhances the paper’s value, fostering further research and collaboration within the scientific community.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • The paper lacks related work concerning routing techniques, which is essential for providing context and understanding the novelty of the proposed approach. • Main results are relegated to the supplementary material, which diminishes their visibility and accessibility to readers. • References to supplementary material are overly vague, making it difficult for readers to locate specific information. For example, statements like “This process does not require any supervision, yet we demonstrate in the supplement that the learned instructions IIR are task-relevant” lack specificity and clarity. How is that shown? Where is that shown exactly? • Visual example can only be found in supplemental material.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    • The related work section could benefit from providing more detailed information, such as quantifying the interference metric as defined in previous research. Additionally, statements like “the task interference issue is rarely explicitly addressed by current all-in-one methods” may suggest a misunderstanding or oversight regarding existing literature.

    • Figure 1 (Overview of the proposed all-in-one medical image restoration (AMIR) network) is not adequately mentioned in the text. Also more information in the caption could help the comprehension.

    • Discrepancies between the text describing the comparative experiments and Table 1 are irritating This section should be structured differently and the results should appear in the paper. The authors should first list the methods that are used in the experiments. And mention and describe the experiments in the suppl. in a separate step.

    • The absence of discussions on relevant approaches like DRCM and AIR Net in the related work section weakens the paper’s contextualization within the existing literature.

    • The paper lacks a discussion section, which is crucial for interpreting the results, addressing limitations.

    • I was wondering if just retraining the best performing baseline model for the all-in-one task might be a little bit unfair, since this model might be too specialized for the original application?

    • The result tables are missing a standard derivation for a better interpretation of the results.

    • The summary should not be included in one of the ablation study experiments.

    • I was wondering how the different amount of data is handled? Or if that is considered at all? 578 vs 10 vs 159 images are quite uneven in terms of representation. I believe that especially during training, sampling strategies should be used to make it more even for the different applications. Unfortunately I was not able to find any information or discussion about that.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a relevant task with a novel approach and convincing results. However, it should only be accepted if main results are included in the main paper rather than relegated to supplementary material. Additionally, providing more context about related work and clarifying the novelties of the paper would benefit the reader.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The authors propose AMIR, a novel All-in-one Medical Image Restoration (MedIR) network, akin to a U-Net architecture, incorporating a task-adaptive routing strategy. This strategy directs conflicting task inputs along distinct network paths, mitigating interference across various image restoration tasks. AMIR encompasses routing instruction learning, spatial routing, and channel routing. The authors validate their approach on three MedIR tasks: MRI superresolution, CT denoising, and PET image synthesis.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    • Simplicity: The paper is commendably straightforward and easy to grasp. • Extendability: The method’s versatility suggests its potential applicability to a range of other MedIR tasks, bolstering its credibility. • Results: Extensive experiments showcase impressive outcomes. • Method: Introducing CRM learning, a soft-binary mask in the decoder path to counter task interference, is an intriguing addition.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • Limited Comparison: The absence of inference time comparisons with baseline methods, crucial in MedIR evaluations, is a notable gap. Additionally, the linear increase in parameters and training time due to Spatial Routing Modules warrants attention. • Method: Lack of clarity regarding the refinement of deep features (I^{DF}) using Transformer blocks, as these features are not illustrated in Fig. 1. An ablation study of AMIR without utilizing refined features (I^{RF}) is imperative. • Minor Issues: 1. Including visual comparisons in the main paper, with at least one example from each task, would enhance clarity. 2. Exploring the network’s performance with MRI images combined from different downsampling factors (e.g., 8x, 16x) during training would be beneficial.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    • Comments: A more thorough exploration of ablation studies concerning the combinations of training tasks, particularly for MRI Super-Resolution, would enhance understanding. Clarification on why AMIR is more effective when trained with the CT Denoising task is necessary. • Future Recommendations: 1. Extension to 3D: Given the architecture’s simplicity, extending the approach to 3D MedIR tasks is promising. 2. Diversified Performance Evaluation: Exploring performance across additional challenges, such as sparse-view/limited-angle CT, would enrich the evaluation.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    While the clinical relevance of this paper is clear, the need for addressing the comparison gap by including inference time comparisons, clarifying method details regarding feature refinement and visual representation, and addressing minor issues such as visual comparisons, all of which would enhance the clarity and reliability of the paper, strengthening its contribution to the field.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We sincerely appreciate the reviewers for acknowledging our methodological contribution and providing constructive comments for further clarification. Our feedback is as follows. Q1(R1) Limited CT dataset. A1: We follow the approach of CTformer and Eformer by using the AAPM dataset, which includes approximately 400 images per patient, each with a resolution of 512×512. In the future, we plan to conduct our experiments on a larger CT dataset.

Q2(R3) Lack of related work. A2: Routing techniques are predominantly associated with the field of dynamic neural networks [1]. These techniques often involve a routing network that dynamically selects network paths for the main network. Both SRM and CRM are variants of this approach. Our primary contribution is the introduction of a routing instruction network that generates task-relevant instructions and effectively applies SRM and CRM to address task interference in the all-in-one medical image restoration task. We will add this reference in our final submission. Additionally, the interference metric, DRMC, and AirNet will be addressed in the final submission. [1] Han Y, Huang G, Song S, et al. Dynamic neural networks: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(11): 7436-7456.

Q3(R1) Statistical significance. A3: We conduct significance tests, and all our improvements over the comparison methods are statistically significant (p < 0.05).

Q4(R1) Computational complexity. A4: We measure computational complexity on 256×256 images. The computational complexity test results are as follows: AMIR: 127.0573 GFLOPs, Restormer: 141.5756 GFLOPs, and AirNet: 301.0390 GFLOPs. The inference time comparison results are: AMIR: 0.0882s, Restormer: 0.0721s, and AirNet: 0.1430s. These results demonstrate that AMIR has superior computational efficiency and a comparable inference speed to state-of-the-art approaches.

Q5(R3) Main results and figures in the supplement. A5: I will move these results and figures to the main section of the paper in the final submission.

Q6(R3) Vague reference to supplement. A6: We will revise this in the final submission.

Q7(R3) Figure 1 is not adequately mentioned and lacks caption. A7: I will revise this in the final submission.

Q8(R3) Only the training the best performing baseline for all-in-one task is unfair. A8: I will retrain other baselines for all-in-one tasks in future studies.

Q9(R3): Concerns about the sampling strategy. A9: We currently use a random sampling strategy. We will conduct ablation studies on different sampling strategies in our future research.

Q10(R4) Issues regarding I^{DF} and I^{RF} A10: We will make a clear annotation of I^{DF} in the final submission. And we will make ablation studies on I^{RF} in further studies.

Q11(R4) Exploring more downscaling factors during training MRI SR. A11: It is a good suggestion and we will take it into consideration in our future work.

Q12(R4) Further experiments on task combinations. A12: We will explore our algorithm with different task combinations as more training tasks are included. Generally, one task may positively or negatively impact others based on their similarities and how the model handles their relationships. In our experiments, the CT denoising task has shown benefits to other tasks, but this requires further exploration.

Other issues will be addressed in our final submission.



Meta-Review

Meta-review not available, early accepted paper.



back to top