List of Papers Browse by Subject Areas Author List
Abstract
Accurately localizing two-dimensional (2D) ultrasound (US) fetal brain images in the 3D brain, using minimal computational resources, is an important task for automated US analysis of fetal growth and development. We propose an uncertainty-aware deep learning model for automated 3D plane localization in 2D fetal brain images. Specifically, a multi-head network is trained to jointly regress 3D plane pose from 2D images in terms of different geometric transformations. The model explicitly learns to predict uncertainty to allocate higher weight to inputs with low variances across different transformations to improve performance. Our proposed method, QAERTS, demonstrates superior pose estimation accuracy than the state-of-the-art and most of the uncertainty-based approaches, leading to 9% improvement on plane angle (PA) for localization accuracy, and 8% on normalized cross-correlation (NCC) for sampled image quality. QAERTS also demonstrates efficiency, containing 5× fewer parameters than ensemble-based approach, making it advantageous in resource-constrained settings. In addition, QAERTS proves to be more robust to noise effects observed in freehand US scanning by leveraging rotational discontinuities and explicit output uncertainties.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2943_paper.pdf
SharedIt Link: https://rdcu.be/dVZeG
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72378-0_39
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/2943_supp.pdf
Link to the Code Repository
https://github.com/jayrmh/QAERTS.git
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Ram_Geometric_MICCAI2024,
author = { Ramesh, Jayroop and Dinsdale, Nicola and Yeung, Pak-Hei and Namburete, Ana I. L.},
title = { { Geometric Transformation Uncertainty for Improving 3D Fetal Brain Pose Prediction from Freehand 2D Ultrasound Videos } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15001},
month = {October},
page = {419 -- 429}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution of the paper is a pose estimation method (QAERTS). It regresses the 3D position of planes in space from freehand 2D ultrasound videos. The target anatomy is the fetal brain.
- The pipeline incorporates various elements from multi-head components to regress 3D locations. Regress diverse geometric transformations, variability, multi-task uncertainty and geometric projection.
- Extent uncertainty based approaches.
- Comparison with baselines and empiric validation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A novel method for localization in space of 2D freehand ultrasound videos.
The main inspiration of this paper is the method “Learning to map 2D ultrasound images into 3D space with minimal human annotation” by Pak-Hei Yeung et. al. An extension is proposed by incorporating multi-head components to regress the 3D locations. Notably the ensemble methods are included in the analysis. Other approaches are extended and used as baselines for comparison.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Relevant work to this paper was not cited including “Deep learning-based plane pose regression in obstetric ultrasound” by Chiara Di Vece et. al. published in 2022 and his more recent work “Ultrasound Plane Pose Regression: Assessing Generalized Pose Coordinates in the Fetal Brain” published in 2024.
Both papers focus on localization of 2D planes in 3D space. The 2022 paper introduces the deep learning method and demonstrates the application in a phantom as well as real world data of the fetal brain. The 2024 paper builds on this by applying the model to real-world data and demonstrates the potential for practical applications in obstetric ultrasound scenarios.
“Long-Term Dependency for 3D Reconstruction of Freehand Ultrasound Without External Tracker” by Qi Li et. al. primarily focuses on the development of a method for reconstructing 3D ultrasound imagery without the need for external tracking devices. This paper is relevant for the task in question as well and was not properly cited.
The best performing method is Deep Ensemble (DE) and there is no detailed description of this model in the paper. For instance, it is not clear at all what models are included in the ensemble. Moreover, the DE method outperforms the proposed method QAERTS. One relevant question to answer in the rebuttal face would be to include QAERTS in the list of DE methods. Does including QAERTS improves the DE prediction?
The NN architecture is also similar to Pak-Hei Yeung et. al. The contributions made to this architecture must be clearly highlighted.
The real-world data test set is severely limited as it is only comprised of 3 subjects. Moreover, the results table does not differentiate between the 2 test sets.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Do you have any additional comments regarding the paper’s reproducibility?
The code will be made public upon acceptance of the paper.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The paper would benefit from a more detailed description of the convolutional neural network (CNN) architecture used. There is no information about the feature extraction model and the pink/blue boxes in the figure. What are these layer configurations or activation functions?.
Including more information about the data preprocessing steps and the augmentation techniques could enhance the replicability of your results. Details on how the datasets were compiled, and any image normalization or transformations applied, would be useful.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The results presented are not convincing enough. The proposed method QAERTS does not outperform the DE method. It is not clear what models are part of the ensemble and it makes the evaluation of the contributions challenging.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper introduces QAERTS, a multi-head model for 3D fetal brain pose estimation from freehand 2D ultrasound videos. The suggested model is demonstrated to outperform all baselines with similar parameter size.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed method is novel and technically sound.
- The experiments are abundant and through. The results are promising.
- The writing is good.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My main concern lies on the application of this method in clinics.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
This is a good paper with clear writing/illustration, method design, and experiment support. My main question is about the application of the proposed approach in the clinics - how would the sonographer use the model? In addition, the statements on the parameter efficiency paragraph on page 8 might be imprecise - the inference latency is correlated, but not linearly correlated to the model parameter size. Besides, could the authors explain why the standard deviation in Table 1 is that high? Furthermore, would random scaling in data augmentation affect the ground truth pose?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper does not have obvious technical flaws. My main concern is its real-world application.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The presents a novel method for estimating the 3D orientation of a 2D ultrasound plane, bu using an ensemble of geometric transformation. The ensemble is used to estimate an uncertainty on the output, and is computationally efficient compared to a deep emsemple.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper clearly lays out a novel light weight ensemble to robustly estimate the 3D pose of an image with more consistent results. The lightweight nature of the model makes it more suited for potential application in low income settings.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The performance of the model - while better than PlanInVol - is clearly subpar to the deep ensembles model also formulated by the authors. Potential avenues to bridge this gap in performance would be a nice contribution to the discussion.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
The supplementary material is sufficient, but could contain more examples from the edge cases in the testset.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
I don’t see how the second paragraph follows from the first: “Instead of assuming the aleatoric uncertainty to be the same for all the data samples (homoscedastic), we explicitly model the noise considering heteroscedasticity. This is because subjective sonographer judgment and potential fetal motion relative to the probe placement [20] is likely to affect each frame in 2D US scans differently.” How will the sonographer judgment affect the images - are you referring to expertise level?
Typo in the beginning of baseline models: “aporoach”
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Pose estimation in ultrasound is an important topic with many downstream application - especially in low income settings. I find the technical description and results convincing.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We would like to thank all the reviewers for their constructive comments and suggestions. We have grouped our responses by topic.
Novelty/Contribution: Response to R1 (a) The 3 suggested papers relevant to the background of this study will be added to the introduction. (b) We will clarify in the paper that the individual models included in the ensemble DE are independent MVE models, which is further described in Section 2, Page 4. (c) As one of the objectives of our work is to be operational in resource-constrained settings, we focus on the parameter efficiency of QAERTS. We did find that adding QAERTS components to DE does slightly improve the performance, however not substantial (less than 3% improvement across any metric). We will add this set of experiments to the Supplementary file. With DE, the explicit ensembling already contributes to this, so the implicit ensembling through QAERTS is more beneficial for non-ensembled models. We have included this explanation in Section 4, Page 9, under the sub-heading Loss Landscape. (d) The modifications to the backbone architecture/feature extractor presented in [26], activation function used and adaptations to the training process are now expanded in Figure 1 caption and where applicable in Section 2, specifically under the sub-headings: Parameterizing Rotational Representations, Uncertainty-Aware Learning and Modifying loss. Configurations regarding data preprocessing and augmentation to improve replicability have been added to Section 3, Page 5. We now emphasize the three main adaptations to the baseline proposed as: a) enabling the model to regress both a mean and standard deviation, b) expanding a single output head into multiple with each regressing means and standard deviations of different geometric transformations, and c) changing the loss function from mean squared error to a negative gaussian likelihood loss during training. The Supplementary file further reports ablations performed to validate the benefit of each adaption.
Clinical Applicability Response to R4 (a) Sonographer use: a. Our focus is on assisting novice sonographers and enabling scanning guidance with point-of-care probes in resource-limited settings. This is achieved by improving the 3D localization performance as it allows for better identification of standard planes and interpretation of anatomical landmarks in the fetal brain. (b) Parameter efficiency: a. The proposed approach only requires 41k more parameters than the baseline model to attain a considerable improvement in performance without a significant overhead in computational cost compared to DE, which requires 100M more. Hence, in our specific scenario, we believe that large difference in parameters does affect both the latency and memory. (c) Table 1 result: a. The std is likely high due to the large range of potential values for both the 3D coordinates (reflects in ED, PA and MSE), and the non-normalized pixel values in the sampled 2D US frames (reflects in NCC and SSIM). In small structures such as the fetal brain, minor misalignments between ground truth and predicted values can cause exaggerated differences with some locations/frames. (d) Data augmentation: a. Random scaling was performed on a sample-by-sample basis. For instance, the images were sampled from a 3D location first, then scaled, allowing for robustness to invariance caused during the randomness of US acquisition.
Improvements and Future Work Response to R5 (a) Two potential avenues to improve our proposed work have been incorporated into the Discussion under the sub-heading Future Work. (b) Yes, we are referring to the role of expertise level in locating standard planes, and this clarification has been added to the manuscript as well. (c) The manuscript has been proof-read and corrected for typos and grammatical errors thoroughly as suggested.
Meta-Review
Meta-review not available, early accepted paper.