List of Papers Browse by Subject Areas Author List
Abstract
This paper investigates both biomechanical-constrained non-rigid medical image registrations and accurate identifications of material properties for soft tissues, using physics-informed neural networks (PINNs). The complex nonlinear elasticity theory is leveraged to formally establish the partial differential equations (PDEs) representing physics laws of biomechanical constraints that need to be satisfied, with which registration and identification tasks are treated as forward (i.e., data-driven solutions of PDEs) and inverse (i.e., parameter estimation) problems under PINNs respectively. Two net configurations (i.e., Cfg1 and Cfg2) have also been compared for both linear and nonlinear physics model. Two sets of experiments have been conducted, using pairs of undeformed and deformed MR images from clinical cases of prostate cancer biopsy.
Our contributions are summarised as follows. 1) We developed a learning-based biomechanical-constrained non-rigid registration algorithm using PINNs, where linear elasticity is generalised to the nonlinear version. 2) We demonstrated extensively that nonlinear elasticity shows no statistical significance against linear models in computing point-wise displacement vectors but their respective benefits may depend on specific patients, with finite-element (FE) computed ground-truth. 3) We formulated and solved the inverse parameter estimation problem, under the joint optimisation scheme of registration and parameter identification using PINNs, whose solutions can be accurately found by locating saddle points.
The first experiment shows that the differences between linear and nonlinear models were not found statistically significant for registration but their respective benefits may depend on specific patients. The second experiment demonstrates that the nonlinear model exhibited evident advantages over linear counterparts (p=0.002) in predicting ratios of tissue stiffness between two sub-regions of the prostate.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0208_paper.pdf
SharedIt Link: https://rdcu.be/dV1PZ
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_53
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0208_supp.pdf
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Min_Biomechanicsinformed_MICCAI2024,
author = { Min, Zhe and Baum, Zachary M. C. and Saeed, Shaheer Ullah and Emberton, Mark and Barratt, Dean C. and Taylor, Zeike A. and Hu, Yipeng},
title = { { Biomechanics-informed Non-rigid Medical Image Registration and its Inverse Material Property Estimation with Linear and Nonlinear Elasticity } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15002},
month = {October},
page = {564 -- 574}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a learning-based biomechanical-constrained non-rigid registration method using PINN (physics-informed neural network) where linear elasticity is generalised to the non-linear version. In the experiments, the paper compares the linear version to the non-linear version and demonstrates that the non-linear version has similar registration performance while better parameter estimation ability.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper extends the linear elasticity estimation to non-linear elasticity estimation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The proposed method in the paper is very similar to the method in [14], e.g. the almost same pipeline (see Fig.1 in [14]), the same loss terms (see Eq.1 in the paper and Eq.10 in [14]).
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
It is highly recommended to clarify the difference of the paper in comparison to [14] and highlight the originality and contribution of the paper compared to [14].
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Reject — should be rejected, independent of rebuttal (2)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper shares to many similarities with [14] and does not demonstrate enough originality and technical contributions.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This paper introduces physics informed NNs to regularise poinset image registration and estimate material properties using non-linear elasticity. Formulated as a forward registration process, and inverse (parameter estimation), which offers the potential for in-vivo estimation of tissue properties.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The formulation is principled in that embeds more sophisticated physical material properties into a registration model - the spatial derivatives of the material parameters are estimated pointwise by a by a “Stress branch” of the model. This formulation offers a nice way of baking physical knowledge into the problem. The inverse model formulation offers a unique selling point, perhaps this also improves the biological plausibility of the transformations? even if the RMSE is not improved?
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The presentation of the loss equation (eq 1) is a bit confusing as NN parameters are included in here as well as the whole dataset, but not the Lamé parameters. This, and the lack of the Lamé parameters in Figure 1, makes it a bit tricky to follow the explanation of what information is used where in the calculation of the loss, particularly with an extra learnable function $\Beta_k$.
Presumable there is strong prior information that could be incorporated into these material parameter estimates? Can this be easily incorporated? There’s no discussion of this. There’s also no discussion about variability due to initial material properties.
The experiments could do with some elaboration, particularly on the inverse problem estimation, Figure 4 in particular is a bit hard to interpret - does the Youngs modulus ratio always collapse? Is there a confidence interval if Fig 4 - it’s hard to see and not explained what that’s over.
It would be nice to have a discussion about why points are preferred over image information for this formulation?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
It’s quite tough to follow exactly what the authors have done due to the choice of notation and the incomplete diagrams.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
I think figure 1 could be split into two, a high-level one and lower-level image that explains exactly what the stress prediction branch outputs and how this fits into the loss function. The mathematics could be clarified as described above.
Some of the mechanical details could be relegated to the appendix to make more space for the explanation of how these fit into the model.
Minors: As TRUS is not used/discussed in the paper. Remove from figure 1 as it confuses the explanation of the experiments.
Section 2.1, lame -> Lamé
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the overall idea seems like a good one, there are some issues with the presentation of the method that make it hard to be confident in understanding precisely how the method works.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
The paper offers a technical contribution and the rebuttal offers some clarifications, but the highlighted issues related to presentation that would need to be checked if this paper was accepted.
Review #3
- Please describe the contribution of the paper
This paper presents a PINN framework for biomechanically constrained non-rigid point set registration and the inverse problem of estimating Young’s modulus value.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Detailed comparison was made between setup of w/ vs. wo PINN, and linear vs. nonlinear modelling.
- The proposed nonlinear PINN gives estimation of tissue property significantly better than the linear counterpart.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- As mentioned by the authors, the first weakness is that the experiments are limited to synthetic datasets with small data sizes.
- The saddle point detection procedure for estimating the Young’s modulus ratio was not presented, and its results seemed unstable on the synthetic data.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- The detailed procedure of locating saddle points during the training process should be presented.
- I wonder is it a common practice to estimate the ratio of the Young’s modulus values and what are its clinical values in prostate diagnosis?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper presents a nonlinear PINN for point-set registration and parameter estimation on synthetic prostate data. However, some details on the experiments and motivations are not very clear.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
R1: Reviewer#1 We appreciate that this paper is technically dense and we have made every effort to clarify the details. We believe all questions from R1 can be addressed without significant changes to the paper and we summarise them as follows. Feedback to the main weakness. (1) The dataset is considered to include the information of Lamé parameters. Lamé is defined in Sec. 2.1. We will clarify that Lamé is part of the Eq. (1) and add Lamé in Fig. 1.
(2) In the inverse problem of estimating material properties, we adopt the clinical-relevant assumption that PZ and TZ regions generally exhibit different stiffness properties. This assumption has been described in Sec. 2.2 (the third sentence of this paragraph), and has been incorporated (easily) into Eq. (2) (i.e., beta_k). (3) Table 1 includes more information, and shows that the nonlinear model can succeed in all test cases while the linear model fails in 1/3 cases, which has been described at the end of Sec. 3. The confidence interval is represented as shadow area in Fig. 4, which will be briefly introduced in the final version. (4) The choice of points over images.
Most previous work (e.g., [14]) mostly used point clouds rather than image intensity directly for several practical constraints, such as the large number of voxels and unclear benefits in using intensity values for modelling biomechanics. In addition, we note that point clouds as robust features that can be directly extracted from raw medical images (e.g., voxels) in two spaces, are very suitable for multi-modality registration. In contrast, there is still debate what the most suitable similarity metrics are to be used for the multi-modal medical image registration (i.e., the MRI-US registration in this study). This point will be added.
Feedback to the constructive comments. The organization, description (e.g., mathematics, elaborations of experiments, the variability), and figures will be revised as requested without significant changes in the final version.R3: Reviewer#3
R3 is mainly concerned about the contributions compared to [14].
We clarify that the main new contributions include (1) biomechanical constraints are generalised from linear to nonlinear elasticity; (2) the inverse problem (and its solution) of estimating material properties with soft tissues is formally formulated using a completely different objective with [14]; (3) the influences of the nonlinear elasticity over performances of solving both forward and inverse problems have been explored. We should note that formulations in governing equations differ significantly from those in [14] as second-order terms are considered, and the formulation of the inverse problem is entirely new (using both linear and nonlinear elasticities). Besides, Eq. (1) in this paper is a general framework for the forward problem, where both linear and nonlinear elasticities can be accommodated. On the other hand, we politely point out that codes will be released as stated in the Abstract.R5: Reviewer#5
(1) R5 is concerned about the saddle point detection procedure, which is done by finding the flat line in Fig. 4. We assume that “saddle point” itself can convey this information, but this point will be clarified. (2) Estimating the ratio of the Young’s modulus between two regions plays an important role in biomechanics-constrained non-rigid point set registrations. This is verified in previous published work [a]. We will emphasize this point in the final version. (3) R5 also asks the clinical value of estimating the ratio of the Young’s modulus. First, the ratio of Young’s modulus is essential for the biomechanical-constrained non-rigid registration, which is used in the ultrasound-guided prostate biopsy procedure. In addition, we note that a region with higher stiffness (e.g., Young’s modulus) may have a higher probability of being cancerous. [a] Modelling prostate motion for data fusion during image-guided interventions, TMI 2011.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
2 reviews have converged to acceptance. One reviewer is claimg similarity with 14, but the paper is sufficiantly different.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
2 reviews have converged to acceptance. One reviewer is claimg similarity with 14, but the paper is sufficiantly different.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A