List of Papers Browse by Subject Areas Author List
Abstract
Physics-inspired regularization is desired for intra-patient
image registration since it can effectively capture the biomechanical characteristics of anatomical structures. However, a major challenge lies in the reliance on physical parameters: Parameter estimations vary widely across the literature, and the physical properties themselves are inherently subject-specific. In this work, we introduce a novel data-driven method that leverages hypernetworks to learn the tissue-dependent elasticity parameters of an elastic regularizer. Notably, our approach facilitates the estimation of patient-specific parameters without the need to retrain the network. We evaluate our method on three publicly available 2D and 3D lung CT and cardiac MR datasets. We find that with our proposed subject-specific tissue-dependent regularization, a higher registration quality is achieved across all datasets compared to using a global regularizer. The code is available at https://github.com/compai-lab/2024-miccai-reithmeir.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/3303_paper.pdf
SharedIt Link: https://rdcu.be/dV1P0
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_54
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/3303_supp.pdf
Link to the Code Repository
https://github.com/compai-lab/2024-miccai-reithmeir
Link to the Dataset(s)
https://learn2reg.grand-challenge.org/Datasets/
https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html
BibTex
@InProceedings{Rei_DataDriven_MICCAI2024,
author = { Reithmeir, Anna and Felsner, Lina and Braren, Rickmer F. and Schnabel, Julia A. and Zimmer, Veronika A.},
title = { { Data-Driven Tissue- and Subject-Specific Elastic Regularization for Medical Image Registration } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15002},
month = {October},
page = {575 -- 585}
}
Reviews
Review #1
- Please describe the contribution of the paper
the paper presents a novel deformable image registration method that learns subject specific parameters (Lame parameters) for linear elastic regularization term. this is done to further account for different tissue types in the body e.g. due change over time. the method uses two networks: one network to learn parameters, and second one which takes local parameters (for each position in the images) to better register two input images. the method was validated on 3 data sets, two of them were 2D, and one of them was 3D. there is modest improvement in terms of Dice overlap for the proposed method.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-> subject specific parameter estimation for each type of the tissue is an interesting idea
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) the major weakness: the elasticity parameter maps are added to both similarity measure and to regularization terms. the paper introduction and the motivation says that the paper presents a new method for ‘regularization’, so why similarity measure is also modified?
2) the results modestly show the advantage of the presented approach. the data used for the experiments doesn’t present changes related to the age (this was example given in the introduction)
3) introduction presents only deep learning based registration models for adaptive regularization.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
NA
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
1) would be good to see results for each component that is made adaptive (similarity measure, regularization terms), and some motivation why to make similarity measure adaptive based on linear elasticity
2) If no imaging data available to show effect of ageing, it would be better to show how displacement changes when using the proposed subject-specific regularization for e.g. lung motion estimation (the currently used data set). for example, does the subject-specific, and tissue-type-specific regularization better estimates sliding motion (see some papers below)
3) there is some done for adaptive regularization using the classic iterative methods Stefanescu, Radu, Xavier Pennec, and Nicholas Ayache. “Grid powered nonlinear image registration with locally adaptive regularization.” Medical image analysis 8.3 (2004): 325-342. Pace, Danielle F., Stephen R. Aylward, and Marc Niethammer. “A locally adaptive regularization based on anisotropic diffusion for deformable image registration of sliding organs.” IEEE transactions on medical imaging 32.11 (2013): 2114-2126. Papież, Bartłomiej W., et al. “An implicit sliding-motion preserving regularisation via bilateral filtering for deformable image registration.” Medical image analysis 18.8 (2014): 1299-1311.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
the proposed method heavily relies on paper [17], and the novelty of the paper is patient-specific parameter estimation
the patient-specific parameter estimation however yield only modest improvement
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper discusses the importance of physics-inspired regularization in intra-patient image registration and highlights the challenge of varying parameter estimations and subject-specific physical properties. The authors propose a data-driven method using hypernetworks to learn tissue-dependent elasticity parameters, improving registration quality without retraining the network. Evaluation on lung CT and cardiac MR datasets shows better results compared to a global regularizer.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper introduces a novel registration algorithm that learns subject-specific linear elastic regularization. It stands out as potentially pioneering this approach, which estimates tissue- and subjectspecific parameters for the physics-inspired regularization of data-driven image registration. The paper conducts extensive experiments to demonstrate the effectiveness of the proposed method. This thorough evaluation provides robust evidence supporting the efficacy of the approach across various datasets and scenarios. registration.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper may lack comparison with other state-of-the-art methods or alternative approaches, making it challenging to assess the superiority of the proposed method against existing registration approaches, such as VoxelMorph, VoxelMorph-diff,and IDIR(https://openreview.net/forum?id=BP29eKzQBu3)
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The paper is well-structured, with clear sections like Introduction, Methodology, Experiments, and Conclusion. However, there are areas where additional clarification would enhance understanding. For instance, Figure 1 is currently straightforward but could be enriched with more informative details. Moreover, a comparison with existing registration approaches would contribute to a more comprehensive validation.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The research work presented is meaningful, offering a novel approach to intra-patient image registration. However, there is room for improvement in terms of clarification and experimental validation. Strengthening these aspects would enhance the overall quality and impact of the study.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The authors propose a deformable image registration framework relying on hypernetworks for the estimation of subject- and tissue-specific hyperparameters of the linear elastic regularizer. This work extends on recent works [17] by introducing hyperparameter maps that allow for tissue-specific regularization, thus adjusting the regularization strength for individual tissues/organs at test time for each new subject. The authors evaluate the proposed approach on 2D and 3D contexts, using publicly available datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength of this paper is the introduction of elasticity parameter maps that allow for tissue-specific regularization. The parameter maps are automatically computed for each subject thanks to a hypernetwork trained on the same subject, but with a global elastic regularizer similar to [17]. For each tissue, an optimization problem is solved efficiently with the trained global hypernetwork, leading to optimal tissue parameters per subject.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Regarding the optimization procedure for finding optimal patient- and tissue-specific parameters, the authors only mention that they used “grid search”. Not only are the details not provided, but also I believe that better approaches for this optimization can be used, as for instance gradient-based approaches, which can potentially find a better solution in significantly fewer evaluations.
For the 2D datasets, the authors use slices chosen rather arbitrarily from 3D image datasets. As a result, the resulting 2D fixed and moving images do not contain the same anatomical information, and therefore, the registration task is not physically meaningful, which is important since the authors propose a physically meaningful regularization.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- Explicitly mention that three types of tissue are considered for all datasets, i.e. C = 3. One understands that only near the end.
- “Comparsion” typo in title for section 3.3
- For the 3D Learn2Reg dataset, a class-wise TRE would be helpful
- In the second paragraph of “Spatially adaptive model”, the segmentation map S is probably a scalar field of integers, rather than reals ?
- A comment on the weight for the first term of the training loss (1) is welcomed. The authors mention that the motivation is to avoid adding a regularization weight, but there are possibly other ways of achieving the same without requiring normalization for lambda and mu, which as mentioned by the authors, turns out to be a barrier for parameter interpretability.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper introduces tissue-specific regularization for deformable image registration in the context of deep neural networks, which to my knownledge is a novelty. The paper is well written and the feasibility of the proposed approach is demonstrated in publicly available datasets.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
As before, I still believe this paper has value, since physically inspired regularization can potentially lead to more meaningful deformation fields, and addressing the problem of tissue-dependent parameters constitutes a contribution.
Although there may still be some methodological weaknesses (physically meaningless 2D registration problem, choice of optimization algorithm), I consider the paper worth of publication in MICCAI.
Author Feedback
We would like to thank reviewers R1, R3, and R4 for their constructive feedback and are grateful that they found that the “optimization problem is solved efficiently with the trained global hypernetwork, leading to optimal tissue parameters per subject” (R1), the “subject-specific parameter estimation for each type of the tissue is an interesting idea” (R3) and that we propose a “novel registration algorithm that learns subject-specific linear elastic regularization [which] stands out as potentially pioneering this approach” (R4). Here, we respond to their comments raised:
1) Lack of comparison to SOTA methods (R4): While we do not extensively compare, we do compare to the HyperMorph framework (based on Voxelmorph), and [17]. Our main aim was to introduce and compare different regularization schemes (diffusion vs. elastic/global vs. spatially adaptive) within one consistent framework. We will clarify this in Sec. 3.3. Setting our work in the greater scope of SOTA methods is important, and we will include that in a future, extended version. 2) Adaptive regularization (AR) in iterative registration (R3): We are well aware that AR is explored in many prior works, including in iterative methods, and would like to clarify how we differ from the works mentioned by R3. While we do not model sliding motion (see below), we do learn the effect of physical parameters on the deformation so that efficient inference can be performed. Also, we explicitly model physical tissue properties. We agree that iterative methods are important, and space permitting, we will include them in the introduction, as well as clarify the differences to our proposed method. 3) Missing evaluation of sliding motion (SM) (R3): As already mentioned above, our work does not address SM. Since the linear elastic regularizer penalizes the tearing of elastic material, it cannot support SM in its current form. We appreciate that incorporating SM when working with lung data is relevant. However, our work at this stage focuses on tissue-specific elasticity without targeting specific anatomies such as the lung, which is why we do not consider SM. We agree that this is important to clarify and will add this in Sec. 4. 4) Weighting of similarity term (R1, R3): Thanks to the comments by R1 and R3, we can see that the weighting is not explained clearly enough in Sec. 2.1. Since the Lamé parameters are the inputs to the hypernetwork, they are normalized. To balance the similarity (sim.) and regularization (reg.) terms accordingly, the sim. term is weighted. We do both analogously to [11]. Since we use an adaptive reg. term, we use voxelwise balancing. Hence we use the weight maps in the weighting of the sim. term. We will clarify this in Sec. 2.1. 5) Novel regularization despite adaptive sim. term (R3): We learn the effect of the regularizer on the deformation since the Lamé parameters are the input of the hypernetwork. Thus, we indeed propose a novel regularization even though the weighting of the sim. term is based on the weight maps of the regularizer. 6) Only modest improvement of Dice (R3): While the proposed regularization only slightly improves the mean Dice score, we want to highlight that the adaptive regularization leads to improvements also for the class-wise Dice and the TRE across all datasets (Tab. 1). On top of this quantitative performance increase, the qualitative results reveal that the proposed approach results in more meaningful deformations, which cannot be captured by the Dice alone.
We also appreciate the respective suggestions of R1 and R3 to explore gradient-based parameter optimization and to evaluate our method on age-related data (although currently, we do not have access to a suitable dataset) and will consider this for future work. Following the suggestion by R4, we will add more details to Fig. 1. We again sincerely thank the reviewers for their time and effort and believe that their feedback will improve the quality of our final manuscript.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
Overall the paper should be accepted. The reviewers had concerns in the first round and they did not come back to adjust their scores, if necessary. It seems like there is sufficient interest and the paper will lead to a nice discussion at the conference.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Overall the paper should be accepted. The reviewers had concerns in the first round and they did not come back to adjust their scores, if necessary. It seems like there is sufficient interest and the paper will lead to a nice discussion at the conference.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
This paper is extremely borderline, but two reviewers recommend acceptance, and the third reviewer did not come back to update their score after rebuttal, so I will recommend acceptance.
The proposed method has a clear justification, and subject-specific regularization is an important topic. That said, the work seems very incremental, and I fail to see why two networks are required (there aren’t that many tissue classes, is it so hard to directly perform a grid search on the spatially adaptive network?).
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
This paper is extremely borderline, but two reviewers recommend acceptance, and the third reviewer did not come back to update their score after rebuttal, so I will recommend acceptance.
The proposed method has a clear justification, and subject-specific regularization is an important topic. That said, the work seems very incremental, and I fail to see why two networks are required (there aren’t that many tissue classes, is it so hard to directly perform a grid search on the spatially adaptive network?).