Abstract

The ability to map left ventricle (LV) myocardial motion using computed tomography angiography (CTA) is essential to diagnosing cardiovascular conditions and guiding interventional procedures. Due to their inherent locality, conventional neural networks typically have difficulty predicting subtle tangential movements, which considerably lessens the level of precision at which myocardium three-dimensional (3D) mapping can be performed. Using 3D optical flow techniques and Functional Maps (FMs), we present a comprehensive approach to address this problem. FMs are known for their capacity to capture global geometric features, thus providing a fuller understanding of 3D geometry. As an alternative to traditional segmentation-based priors, we employ surface-based two-dimensional (2D) constraints derived from spectral correspondence methods. Our 3D deep learning architecture, based on the ARFlow model, is optimized to handle complex 3D motion analysis tasks. By incorporating FMs, we can capture the subtle tangential movements of the myocardium surface precisely, hence significantly improving the accuracy of 3D mapping of the myocardium. The experimental results confirm the effectiveness of this method in enhancing myocardium motion analysis. This approach can contribute to improving cardiovascular diagnosis and treatment. Our code and additional resources are available at: https://shaharzuler.github.io/CardioSpectrumPage

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1314_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1314_supp.pdf

Link to the Code Repository

https://github.com/shaharzuler/CardioSpectrum

Link to the Dataset(s)

https://arxiv.org/abs/2406.01040

BibTex

@InProceedings{Zul_CardioSpectrum_MICCAI2024,
        author = { Zuler, Shahar and Tejman-Yarden, Shai and Raviv, Dan},
        title = { { CardioSpectrum: Comprehensive Myocardium Motion Analysis with 3D Deep Learning and Geometric Insights } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15005},
        month = {October},
        page = {pending}
}

Reviews

Review #1

Please describe the contribution of the paper

The author presents a novel regularization approach to handle aperture problems in medical image registrations.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The author gives a good introduction to the problem and how tangential movements of myocardium is related to the aperture problem.
- The author presents a novel approach of regularization based on sparse displacement fields, estimated via referred work [12].
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper includes small figures that are hard to read.
- Explanation of the synthetic data used in the experiment is limited. How is it generated? How many samples were used for training/testing/validation? How was the distribution of torsion of the different sets? Or was the model optimized for each scan?
- The experiment is only performed on synthetic data. Is there any limitation to performing on real data?
- A short background about the backbone methods, ZoomOut, and ARFlow is missing.
- In many image registration methods, the displacement field is constrained using implicit regularization. In this method, the cost function did not include any regularization of the displacement field. Why not?
- Will the flow loss constraint be beneficial over segmentation-based loss for other image registration problems as well? Why, why not?
- Limited explanation about the hyperparameter settings in the model, like lambdas.
- The result is only based on similarities with the ground truth displacement field. How did the methods perform on other metrics, like image similarity, dice score, and smoothness of the displacement field?
- Missing details on why the model was trained in two parts. The first uses an unconstrained and the second uses constrained loss.
- How were the sparse displacements calculated from the mesh vertices? And what was the accuracy of the ZoomOut model?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Do you have any additional comments regarding the paper’s reproducibility?

The data split is not described by the authors.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

The author explains the difficulty with the aperture problem for image registration and suggests a novel approach using sparse displacement field-based regularization to solve the problem. However, the explanation and performance of the suggested method are limited, and in the list of weaknesses I have stated some concerns and questions.
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

Weak Reject — could be rejected, dependent on rebuttal (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think that the author has identified the problem correctly and suggested an interesting approach to solving the problem. However, I request a more comprehensive explanation to specific choice and suggested procedures from the author. Furthermore, the approach is only evaluated on synthetic data, which I per see do not see any concern about, but I missed the link to how the invention can be used in real cases.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #2

Please describe the contribution of the paper

This paper proposes the use of functional maps to find correspondence between end-systolic and end-diastolic cardiac LV meshes, and then use the displacements at these mesh points as regularizer/constraint to solve for the dense myocardial deformation using optical flow. This was done for CT images. Their approach helps tackle the well known aperture problem of optical flow, which helps them achieve good results in cases where there’s a higher degree of circumferential/non-radial motion.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use of a functional map for this problem space seems like a good idea. I couldn’t find examples of usage of functional maps for cardiac shape matching.

The displacements derived from the LV mesh alignment is used as a regularization/constraint, which improves overall displacement estimation, especially in cases where there’s a large toroidal motion.

Ideas regarding the aperture problem and functional maps are presented nicely and makes the paper enjoyable to read.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

My main criticism is that 1) why isn’t there a comparison with some standard non-rigid free form deformation based registration methods (but not optical flow based) for motion estimation? These methods generally tend to do well for Echocardiographic and CMR data from my knowledge - is CT different? Example of some work 1) classic method: https://www.sciencedirect.com/science/article/abs/pii/S1361841511001605 2) CNN based - https://arxiv.org/pdf/1809.05231.pdf, etc. I understand you can’t possibly compare with all the methods out there, but just curious. Maybe you’ve already compared but not reported here, maybe you have strong intuition why that won’t be a good idea at all, etc.

Another issue is that the segmentation masks seem needed? So, we should assume that there’s a segmentation model that has already been trained that works okay?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Do you have any additional comments regarding the paper’s reproducibility?

Code provided as supplementary material. Didn’t check it but looks okay with setup.py available for installations.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

The data part isn’t very clear. It says 300 pairs of volumes used. How can you train your model with only 300 pairs? Also, even though it is mentioned that these are synthetic images, we don’t know much else about them.

Is the method applied only for the registration of ED/ES frames? Typically, registration is required for the entire cardiac cycle, in these kind of situations. Any comments on this? Is there an easy way to extend the solution/optimization to the entire cardiac cycle?

We don’t see good example images of the data (only tiny images in figure 2) or the resulting displacements/strains. I know there isn’t a lot of space. You could potentially shorten your discussion on the aperture problem/optical flow. While very interesting, it may be taking up some valuable real estate.

Do you have some more reasoning on why your model struggles a bit when the torsion angle is low?

The ZoomOut portion of the network is deterministic right? I.e., there’s no backpropagation to it?

If you computed strains, would your method have more reliable circumferential/longitudinal strains?
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

Accept — should be accepted, independent of rebuttal (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Some weaknesses here but overall its a nice idea. The functional map based regularization could be an influential idea for Cardiac image registration and could be added to other registration methods as well.
Reviewer confidence

Very confident (4)
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Review #3

Please describe the contribution of the paper

The authors present a novel method for complex 3D myocardial motion analysis. Specifically, the proposed model brings in global constraints into a 3D deep-learning architecture, based on ARFlow, by exploiting spectral correspondences computed from 2D surfaces (systole and diastole) using ZoomOut. The presented method, trained and evaluated on a synthetic dataset, performs significantly better at capturing tangential movements compared to two baselines, especially in scenarios with increased torsion angles.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The manuscript is extremely well written and accessible since it provides a concise but complete introduction to motion analysis and the aperture problem, it progressively introduces and motivates every component of the proposed pipeline, and perfectly explains all graphics and equations in text. Moreover, the presented method is novel in that it leverages surface-based constraints to improve the performance of the underlying deep-learning-based model. Despite solely training and evaluating the proposed method on synthetic data, the in-depth results are convincing and suggest a significant step forward in overcoming the aperture problem. Finally, the authors have provided extensive supplementary material, including source code and another paper explaining the data generation process.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
From my point of view there are only few minor weaknesses, which could be easily addressed in the rebuttal phase:
- The method is only trained and evaluated on synthetic data, though I understand that an evaluation on real data is not trivial.
- Even though the authors have taken a lot of effort to write a very clear and detailed manuscript, I believe that the uninformed reader would greatly benefit from a very very brief introduction to Functional Maps, the ZoomOut method, and ARFlow.
- Moreover, I believe that the discussion could be extended. Specifically, I would be interested to learn more about potential failure modes of the proposed algorithm. Would it be possible to use an arbitary volume during the systole or diastole as input or does it require information at end-systole/diastole? Would the method work equally well with healthy and diseased hearts?
- The subfigures of Fig. 3 are too small. It may be worth to either reduce the caption to increase the room for the graphics or, alternatively, to move parts of the figures (and the associated text) into the supplementary material.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Do you have any additional comments regarding the paper’s reproducibility?

I appreciate that the authors have added the source code to the supplementary material.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- Section 4.1: since d is an array of 3D displacement, i.e., a tensor, it should be a capital letter and bold
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

Accept — should be accepted, independent of rebuttal (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents a novel method in a very clear and concise way. The evaluation, despite being on synthetic data, is convincing. The minor weaknesses listed above are easily addressible in the rebuttal and would not change the contribution of the paper.
Reviewer confidence

Somewhat confident (2)
[Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

N/A
[Post rebuttal] Please justify your decision

N/A

Author Feedback

We thank the reviewers for their constructive comments and are pleased they find our work novel, significant, convincing, and well-presented. We now address the main issues. R1,R4: Note that in the submission’s supplementary material, we also attached the full soon-to-be-published chapter on the technical details of the data generation process, as we indicated in Chapter 5. We will emphasize this paper’s availability in the revision. R3, R4: Evaluation based on synthetic data: Our experiments utilized real case data with realistic synthetic deformations. As always, the challenge lies within motion validation. MR tagging is not dense enough in the thru-slice direction, and for CT, ground truth data for tangent movement is lacking. Therefore, we generated movement for validation purposes. R1,R3: Applying method to arbitrary timesteps: Our method can be applied to any timestep in the cardiac cycle. We will clarify this in the revision. R1, R4: Model training: Our model was pre-trained on a large motion-based dataset and fine-tuned individually for each of the 300 pairs, as described in Section 4.2 of our manuscript. We will clarify this in the revision. R1,R3,R4: Figure clarity, math formatting, missing introduction to FMs, ZoomOut and ARFlow: We will incorporate all the suggestions in the revision. Thank you for bringing this to our attention. R1,R4: ZoomOut model accuracy, model’s reduced performance on lower angles: We conducted our evaluations in the voxel space. The average mEPE of the constraints from ZoomOut, including errors from converting voxels to mesh and back, was 3.7 mm, affecting all torsion angles, but relatively more at lower angles. We will add the constraints mEPE plot to the supplementary material. R4: Regularization of the displacement field: Our current approach outperformed other methods without standard regularization. We agree that a more sophisticated approach could further improve and is worthy for a follow up. R4: Missing Hyperparams: Hyperparams detailed in Section 4.4. R3: Method’s performance on healthy and diseased hearts: Our motion model is learned using a geometric preserving constraint on the surface, and not on pre-defined parameters of a movement model. As such, it can cope with unseen large deformations. By construction it can handle healthy and diseased hearts. R3: Potential failure modes: We can generate synthetic deformation with multiple solutions based on symmetries where our approach can fail. However, due to the geometric structure of the LV, there is no rotational or reflective symmetry structure, so this is hypothetical in our study case. R4: How to calculate sparse displacements from mesh vertices: conducted by subtracting corresponding vertex pair locations. We will clarify this in the revision. R1: comparison to FFD methods: We prioritized SOTA optical flow methods and selected ARFlow as our baseline. VoxelMorph uses a U-Net architecture, while we use a more recent one. However, we compare to segmentation-based loss similar to VoxelMorph and will add the missing citation. TDFFD and similar B-spline-based methods are tailored for sequences, whereas our method addresses motion between two specific timesteps. R1: Assessment via strains: The choice of window size for strain calculation impacts the trade-off between accuracy and robustness. Since we can’t validate it directly, we did not add it. R4: Performance on image similarity, dice score, and displacement field smoothness: Standard optical flow metrics may not gauge our method’s performance accurately, especially for tangential movement. Methods relying on radial displacement score higher on image similarity and dice but miss tangential accuracy. We prioritize precise surface motion, resulting in 8.7% higher MSE image distance and 13.6% lower dice score compared to the baseline. Radial displacement methods produce smaller displacement magnitudes and smoother fields; our focus on tangential motion increases L1 smoothness by 48.5%.

Meta-Review

Meta-review not available, early accepted paper.

back to top

CardioSpectrum: Comprehensive Myocardium Motion Analysis with 3D Deep Learning and Geometric Insights

Author(s):