Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Structure inconsistency is the key challenge in registration of brain MRI between pre-operative and follow-up phases, which misguides the objective of image similarity maximization, and thus degrades the performance significantly. The current solutions rely on bidirectional registration to find the mismatched deformation fields as the inconsistent areas, and use them to filter out the unreliable similarity measurements. However, this is sensitive to the accumulated registration errors, and thus yields inaccurate inconsistent areas. In this paper, we provide a more efficient and accurate way, by letting the registration model itself to `speak out’ a Noise Removed Inconsistency Activation Map (NR-IAM) as the indicator of structure inconsistencies. We first obtain an IAM by use of the gradient-weighted feature maps but adopting an inverse direction. With this manner only, the resulting inconsistency map often occurs false highlights near some common structures like venous sinus. Therefore, we further introduce a statistical approach to remove the common erroneous activations in IAM to obtain NR-IAM. The experimental results on both public and private datasets demonstrate that by use of our proposed NR-IAM to guide the optimization, the registration performance can be significantly boosted, and is superior over that relying on the bidirectional registration by decreasing mean registration error by 5\% and 4\% in near tumor and far from tumor regions, respectively. Codes are available at https://github.com/chongweiwu/NR-IAM.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1262_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: N/A

Link to the Code Repository

https://github.com/chongweiwu/NR-IAM

Link to the Dataset(s)

https://www.med.upenn.edu/cbica/brats-reg-challenge

BibTex

@InProceedings{Wu_Noise_MICCAI2024,
        author = { Wu, Chongwei and Zeng, Xiaoyu and Wang, Hao and Zhang, Xu and Fang, Wei and Li, Qiang and Wang, Zhiwei},
        title = { { Noise Removed Inconsistency Activation Map for Unsupervised Registration of Brain Tumor MRI between Pre-operative and Follow-up Phases } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15002},
        month = {October},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper addresses the challenge of structure inconsistency in brain MRI registration, which can degrade performance significantly. The authors propose an efficient method, using the registration model to generate a Noise Removed Inconsistency Activation Map (NR-IAM), indicating structure inconsistencies. Experimental results demonstrate that using NR-IAM improves registration performance, reducing mean registration error by 5% and 4% in near and far from tumor regions, respectively, compared to bidirectional registration.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This paper presents an NR-IAM method designed to identify registration-target-specific inconsistencies, aiming to enhance the registration performance of brain tumor MRI between the pre-operative and follow-up phases based on the generated noise-removed mask. The primary novelty lies in the utilization of a reverse activated inconsistency activation map (IAM), if present. Its effectiveness has been demonstrated on both the BraTS-Reg dataset and a privately collected dataset. Comprehensive comparisons with existing registration methods have been conducted, assessing performance across various metrics.it effectiveness has been demonstrated on BraTS-Reg and a private dataset. a lot of comparison has been provided with the existing registration methods, evaluated with different metrics.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    I believe one of the main issues with the current manuscript is its lack of clarity, and the overall English writing needs to be improved. Here are some examples:

    1. On Page 2: “To achieve this, we acquired NR-IAM through the reverse activation of feature maps. However, the resulting map contains noise activations near common structures like the venous sinus. Due to the homogeneity of brain tissue and structure, we then propagate global activations into a target space and subsequently superimpose the global maps to eliminate common erroneous activations.” I think the order of presentation is reversed. The authors should mention the noise removal step before claiming NR-IAM acquisition.
    2. In the first paragraph of Section 2-Methods, NMP should be spelled out with its full name instead of mentioning it in Section 2.2.
    3. The symbols and descriptions in Figure 1 are confusing. For example, in Fig. 1(b), there is an annotation “m=F” which is unclear and inconsistent because here ‘m’ indicates an image, while ‘F’ represents an image set. Additionally, the difference between randomly-paired scans and spatially-paired scans lacks explanation. (4) It seems the step in Fig. 1(a) is an optimization-based approach, and Fig. 1(b) is learning-based? The clarification here seems a little misleading. Please explain more to make it clear, such as “Firstly, we employ a pretrained model in Fig. 1(a) to exploit global activation information. Each spatially-paired scans, mj and fj, are fed into the pretrained model to obtain coarse activation map IAMfj.”
  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    To enhance the clarity and comprehensibility of the manuscript, the authors should provide further clarification, particularly in sections 2.1 and 2.2. Additionally, improving the figures such as Fig. (1) to make the illustrations more comprehensive and accurate would be beneficial. By addressing these points, the clarity and readability of the manuscript would be improved, enabling readers to better understand the methodology and results presented.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    (1) the novelty in this paper seems limited (2) the clarification in this paper is not clear, especially in the method section, and the overall english writing need to be improved

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper proposed a Noise Removed Inconsistency Activation Map (NR-IAM) to indicate structure inconsistencies in pre and post operative brain tumor image registration. It utilizes Grad-CAM to extract structural inconsistency. And it removes noisy activations by using the statistical prior from a group registration of the follow-up images in training set.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The proposed NR-IAM 1) is target for inconsistency map in registration of pre-op and follow up brain tumor MRI 2) remove noisy activations of single inconsistency map by using statistical prior of the training set 3) is evaluated in both public and private datasets and gets lower target registration error

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1) Lack of justification of why Grad-CAM can highlight the inconsistency. The proposed method is based on a strong assumption that the Grad-CAM can accurately locate inconsistency (which is the tumor in the scope of this paper) between moved and fixed images. This might not hold true during the early stage of training when the network is still learning to align the images via the loss and gradient. Besides, from the uploaded anonymous code, the difference between Grad-CAM and the proposed Grad-Inconsistency Activation Mapping (Grad-IAM) is just the order of ReLU activation and weighted summation. I would question that this might be a variant of Grad-CAM. 2) The proposed method is over-engineered with many hyperparameters, e.g., the threshold, the number of follow-up images used to remove the noisy activation, while getting marginal performance improvement. The generalizability to other registration task with abnormalities is unknown, e.g., the liver tumor. 3) A relevant baseline[1] which also address the problem of tumors in brain tumor MRI registration is not included. 4) Lack of visualization to validate that the Noise Mapping Process (NMP) is indeed removing noisy activation. 5) Missing report of increasing amount of training time due to the group-wise registration in NMP process 6) From Table 1,2,3, the significant improvement of the proposed NR-IAM compared to Grad-CAM and DIRAC is the number of non-positive Jacobian determinant. But the author doesn’t visualize the deformation field nor discuss what contributes to a smoother deformation field when using NR-IAM.

    [1] Dong Q, Du H, Song Y, Xu Y, Liao J. Preserving Tumor Volumes for Unsupervised Medical Image Registration. InProceedings of the IEEE/CVF International Conference on Computer Vision 2023 (pp. 21208-21218).

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Anonymized link to the source code is provided. One public dataset and on private dataset are used.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The following points correspond to the aforementioned weakness: 1) Elaborate why the Grad-CAM can highlight the inconsistency (tumor) area and why it is better than extracting with the image dissimilarity loss (e.g. NLCC) 1.1) Elaborate the difference between Grad-CAM row in Table 3 and Grad-IAM row. Is the difference mainly in the order of ReLU and weighted summation.

    2) Include discussion of the generalizability of the proposed method. Include application to liver tumor registration if possible. Show examples where the proposed method failed in addition to cherry-picked good example. It won’t downgrade the method but will be an informative message to the community and follow-up researchers. 3) Include the relevant baseline if possible. 4) Include figures of the IAM before and after NMP, include figures of the noise mask. Discuss the influence of the number of images in the follow-up datasets used in the NMP process. 5) Include the increased amount of training time. 6) Discuss the reason of smoother deformation field and include the visualization, especially in the tumor area. 7) For Table 2, my personal experience is that image-wise similarity is not a good indicator of registration accuracy in longitudinal registration.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This is a fair paper with weaknesses slightly overweighing its merits. With marginal improvement, I would question the necessity of the extra efforts in designing such a complicated process. I wonder if the performance is improved by more accurate masking of the abnormal areas (tumors) or something else. If it is, then I think it should focus more on comparing the masks of different methods visually. If there is something else helping, I think the author has not done a good job of clarifying the contributing factors. Therefore, I give my decision “weak reject”.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    I appreciate the author’s feedback on the visualization. The images of NR-IAM look pretty good and promising. I would still recommend that the author report the extra required time for stages 1 and 2, especially stage 2, and how many images are groupwise registered in stage 2. The overall unsupervised method to extract the tumor and mask it out in image similarity computation is interesting. I will raise my score to “weak accept”.



Review #3

  • Please describe the contribution of the paper

    The paper proposed NR-IAM (Noise reduced inconsistency activation map) to obtain inconsistency maps, important to guide image registration in case of inconsistencies, e.g., due to (dis)appearance of a tumor. Their method involves a method similar to Grad-CAM to obtain the inverse consistency map (IAM) (backpropagating the gradient of the image similarity), followed by noise reduction (NR) (subtraction with an average IAM).
    Incorporating the inconsistency maps in the loss boosted the registration accuracy of a deep learning model (cLapIRN) on brain MRI.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The relevance and contribution of the paper are clear: Inconsistency maps are needed to guide registration models in these inconsistent areas but existing methods for obtaining them have disadvantages. I believe the proposed method is elegant and resource-efficient, considering it does not rely on annotations and the registration model obtains the inconsistency maps itself (meaning no other networks are needed or should be trained to produce the map).
    The experiments are extensive, showing the added value of each method component and comparing to other activation maps including simple (tumor mask) and more complex maps (DIRAC).

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Major: It is unclear how this method would translate to other applications.

    • For example, the method was evaluated on T1-weighted contrast enhanced MRIs on which the tumor is hyperintense, but it is unclear if this hyperintensity is important for obtaining good inconsistency maps. Can the authors elaborate on this? In other words, does the method work on other contrasts?
    • The authors explain (section 2.2) that noise reduction of the inconsistency map is enabled by brain tissue homogeity and dispersed tumor occurences. Are these prerequisites for the method? Can the authors provide a discussion on what type of applications the method may generalize to (e.g., bladder volume differences, gas pockets in rectum, tumors at sites that are less homogeneous)?

    Minor: It is unclear how many landmarks are available in the BraTS-Reg dataset near and far away from the tumor. 6 to 50 landmarks were available but how many were located near and far away from the tumor? Can the authors report more statistics (the median, IQR or mean, std; near versus far away from tumor)?

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    What were the fixed and moving images for the BraTS-Reg dataset?

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Can the authors explain why the proposed method achieves the highest image similarities in the area near the tumor (last row, Table 2)? Given that the image similarity is not optimized in this area, this was not necessarily what I expected.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    See strengths.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Accept — should be accepted, independent of rebuttal (5)

  • [Post rebuttal] Please justify your decision

    The authors have clarified that the proposed method generalizes to other domains (contrasts, anatomical site) and that should be used only on data with homogeneity of structures. The method is elegant and effective in solving one of the more challenging aspects of image registration.



Author Feedback

We appreciate the insightful feedback and suggestions. In response, we have thoroughly considered each comment and carefully respond as follows.

  1. Novelty(R1&3): To overcome image registration with inconsistency, previous works resort to extra segmentation model or bidirectional deformation error. In contrast, our method directly views the hard-to-optimize parts as the inconsistent structures in image registration, which is more efficient and accurate for introducing no extra parameters and inference errors. We believe our work gives a sufficient and effective solution to longitudinal registration, thereby contributing to the community.
  2. Paper writing(R1): We apologize for the misleading symbols and descriptions. We will clarify our manuscript in the final version.
  3. Clarify training strategy(R1&3): Our method contains 3 stages. Stage 1: pretraining with Grad-IAM, which provides pretrained model for the next 2 stages to inference common erroneous activations and model’s initialization, respectively. Stage 2: extracting common erroneous activations with NMP. And Stage 3: using the extracted error maps and image pairs simultaneously to finetune the model. Thus, the NMP process will not increase the training time in stage 3, but it helps to continuously improve our performance in Table4.
  4. Clarify Grad-IAM(R3): We clarify the differences between NLCC loss, Gad-CAM and Grad-IAM as follows: NLCC loss measures the distance across images and is sensitive to small structural changes(Fig.3). Grad-CAM focuses on objects associated with given classes, but is unaware to unsupervised target. An image registration method can be optimized by maximizing the image similarity, but the similarity in inconsistency areas can hardly be optimized. We thus propose Grad-IAM with 2 operations to concerns the areas, that is inverse operation to explore inconsistency information and channel-wise activation to block interference from consistency information. Therefore, as learning deepens, it even can accurately and stably locate inconsistencies in the early training stage (shown in our released link). Here we emphasize that ‘Grad-CAM’ in Table3 is an inversely operated Grad-CAM.
  5. Generalizability(R3&4): We have tested our method under the same setting on t1 and t2 weighted brain MRI and liver tumor dataset. The results released in our code link reveal the promising generalizability of our method.
  6. Add the relevant baseline(R3): We are including the relevant baseline into the comparison.
  7. Visualize IAM, NR-IAM and noise mask(R3): We sincerely appreciate the insightful suggestions, we will add the visualization results and related discussions to final version. Reviewers can refer to our code link for some visualizations.
  8. Why the predicted deformation filed is smooth(R3): Due to the more accurate location of NR-IAM, it can effectively mask the inconsistent areas and avoid chaotic deformation of surrounding areas. Therefore, NR-IAM yields smoother deformation fields over other methods.
  9. Image-wise similarity on private dataset(R3&4): We test image similarity by registering a normal scan onto all tumor scans, and dividing whole brain into 4 separated areas, e.g., tumor, near tumor, far from tumor and background areas. We only measure the image-wise similarity in near and far from tumor areas. Such areas can be optimized and thus achieve higher performance.
  10. Homogeneity assumption(R4): NMP aims to specify erroneous activations of common structures across registered images. It cannot locate error activations without homogeneity.
  11. Statistics of landmarks(R4): We will add the statistical information of landmarks (near tumor: 3.57±2.18, far from tumor: 5.3±2.76) to the final version.
  12. Regarding the fixed and moving images(R4): The fixed and moving images in stage 1 are randomly selected in BraTS-Reg. But the 2 images in stage 2 and 3 are pre-operative/follow up and follow up/pre-operative scans, respectively.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The paper presents an interesting approach to tackle content inconsistency issues in medical image registration. Two reviewers have upgraded their ratings.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The paper presents an interesting approach to tackle content inconsistency issues in medical image registration. Two reviewers have upgraded their ratings.



back to top