List of Papers Browse by Subject Areas Author List
Abstract
Image monitoring and guidance during medical examinations can aid both diagnosis and treatment. However, the sampling frequency is often too low, which creates a need to estimate the missing images. We present a probabilistic motion model for sequential medical images, with the ability to both estimate motion between acquired images and forecast the motion ahead of time. The core is a low-dimensional temporal process based on a linear Gaussian state-space model with analytically tractable solutions for forecasting, simulation, and imputation of missing samples. The results, from two experiments on publicly available cardiac datasets, show reliable motion estimates and an improved forecasting performance using patient-specific adaptation by online learning.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1838_paper.pdf
SharedIt Link: https://rdcu.be/dV1T4
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_66
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1838_supp.pdf
Link to the Code Repository
https://github.com/ngunnar/2D_motion_model
Link to the Dataset(s)
https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html
https://echonet.github.io/dynamic/
BibTex
@InProceedings{Gun_Online_MICCAI2024,
author = { Gunnarsson, Niklas and Sjölund, Jens and Kimstrand, Peter and Schön, Thomas B.},
title = { { Online learning in motion modeling for intra-interventional image sequences } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15002},
month = {October},
page = {706 -- 716}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes an online learning model for cardiac image sequences generation. Both quantitative and qualitative result show the feasible motion estimation by the proposed model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- An online learning model with fewer parameter at training can predict cardiac image at real time.
- The images with limited time sequence or with relative small sampling rate could also benefit from the proposed model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- It is unclear why the Markov process is used for the cardiac motion estimation. Because the cardiac motion is cyclical. However, the state in the Markov process is dependent on its previous one but not on the other states. But the cardiac motion state can benefit from other previous states as well and they are not independent to each other.
- The author claims the architectural improvement in this paper but such improvement is unclear compared to their previous work.
- For the EchoNet-Dynamic dataset, a moving horizon of N=75 is chosen here. How will this parameter affect the overall performance of the model? Some ablation study could be helpful.
- From Table 1, the results compared with other two conventional non-deep learning methods from 15 years ago are have little improvement.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The author can refer to the comments from the question 2.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The overall improvement from their previous work is more focusing on the parameter reduction for online learning. The Markov process applied for cyclic cardiac image sequence prediction is questionable.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
Built on top of the author’s previous work [1], this paper improved the linear Gaussian state space model to capture the spatial-temporal changes of the images with online learning procedure.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The paper is generally well-written and easy to follow. (2) The idea is well-motivated based on the previous work.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
(1) The comparison methods are not appropriate. SyN and elastic registration can not represent the state-of-the-art registration models. They are all over 15 years old. There are tons of registration methods especially with deep learning approaches that are much better than them. (2) The contribution seems incremental. (3) Since this paper is studying the online learning of image deformation, there should be more literature review regarding the longitudinal deformations of image sequences (image regression problem).
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
None
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
(1) It is recommended to provide significance test to compare the results in Table 1. The results are so close. (2) Figure 3 and 4 are hard to see the difference on the left figures. (3) scaling and squaring is not “a proven approach to obtain diffeomorphic registrations”. It has to do with the regularizations.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Based on the presentation, the novelty, and the experiments of this paper, the reviewer suggest weak accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
This paper is in relatively good shape compared with other papers the reviewer has reviewed. The rebuttal has addressed some of the reviewer’s concerns. The reviewer also looked at other reviewers’ comments, and finally decide to stay the same rating: week accept.
Review #3
- Please describe the contribution of the paper
This paper presents a learning-based motion estimation method based on Gaussian latent state space models. The experiments have demonstrated its effectiveness on cardiac MR and US images in terms of imputation and extrapolation, as well as patient-specific adaptation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper proposes a novel and solid generative model to infer the motion transformations, which is able to deal with missing images and extrapolation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- An illustration of the network architecture is not presented in the paper.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- A typo is made in Page 3: “Finally, given an approximate posterior q(x \mid y_0, y )”, where q(x \mid y_0, y) should be q(x, z \mid y_0, y).
- An illustration of the network architecture should be presented in the paper or the supplementary material, especially the decoder architecture.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is novel in methodology and convincing in experiments. I am happy to recommend acceptance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
First of all, we would like to thank all reviewers for their responses and opinions. A common opinion (R3, R4) is that we have not compared our model against DL-based image registration models. Here, we would like to point out several things that could be clarified in the paper: i) our model is not a pure image registration model. The purpose is to estimate missing samples in the image sequence. In the experiment, we want to show that our model shows similar results to two recognized methods in image registration that rely on known observations, and we do not strive to be state-of-the-art in image registration. ii) of course, we could have chosen other DL-based models to compare against. Our choice relies on the work of Krebs, J. et al. 2019. In that study, SyN, one of our comparison methods, shows better results than Voxelmorph (Balakrishna, G. et al. 2018), a recognized DL-based model in image registration. However, a direct comparison against Krebs, J. et al. 2019 is not possible as the source code for that model is not available. But if we compare the result in our experiment with theirs, they are approximately equivalent. This also applies to the comment that the results in the table are “so close” (R3). The purpose of the comparison is not to outperform existing models but to demonstrate the accuracy of the model’s image registration while we can handle interpolation and extrapolation for missing samples.
Another comment we want to highlight is our choice of a Markov process to estimate the motion (R4). Here, we emphasize that Markov processes can capture a wide range of motion types, including cyclic motions (see e.g. Åström and Murray, 2008). The key is that by using a state with dimension higher than 1 we can straightforwardly create various oscillatory motion. We see several advantages of using a linear Gaussian state space model as our dynamical model: i) it is simple, with only a few parameters that contribute to fast real-time training; ii) its inference solutions, such as filtering, prediction, and smoothing, are analytically tractable, and finally; iii) stability analysis is possible by studying only the eigenvalues of the A matrix. The last aspect is not included in the paper, as it would have meant violating the 8-page limit. Furthermore, the Markovian variable is the latent state variable z, not the observed variable x. This contributes to an increased variety of dynamics.
Regarding more references concerning longitudinal deformations for image sequences (R3), we will add a brief discussion of this in the final version. An approach that is not mentioned, for example, is vector momentum-based methods (Yang, Xiao, et al. 2017, Pathan and Hong, 2018, Ding, Zhipeng, et al. 2019), where the models are trained supervised, generating a vector momentum sequence using LDDDM and geodesic shooting as ground truth. Those methods have, as far as we know, only been evaluated on brain sequences and not cyclic patterns, which was our primary target in this paper. Other works using adaptive online learning procedures (Sharp, Gregory C, et al. 2004, Jöhl, Alexander, et al. 2020, Lombardo, Elia, et al. 2022, Li, Yang, et al. 2023) were left out since their models rely on e.g. respiratory signals instead of the image sequences. If the reviewer thinks other references are missing, we welcome suggestions.
During online learning, we choose a moving horizon of N=75 (R4). The length was considered a hyperparameter. Our choice is based on a trade-off between a long horizon to cover the most significant motion in the sequence and a relatively small one to adapt the model fast which could be further clarified and evaluated in the paper.
For the comment that scaling and squaring is not “a proven approach to obtain diffeomorphic registrations” (R3), we agree. What we mean by this sentence is that using Gaussian smoothing and scaling and squaring layers showed diffeomorphic registrations in the reference work [9,17].
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The overall recommendation from the reviewers is positive (WA/WR/A). The authors are highly suggested to carefully incorporate all reviewers’ comments in a revised manuscript.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The overall recommendation from the reviewers is positive (WA/WR/A). The authors are highly suggested to carefully incorporate all reviewers’ comments in a revised manuscript.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A