List of Papers Browse by Subject Areas Author List
Abstract
Intraoperative ultrasound (iUS) imaging has the potential to improve surgical outcomes in brain surgery. However, its interpretation is challenging, even for expert neurosurgeons. In this work, we designed the first patient-specific framework that performs brain tumor segmentation in trackerless iUS. To disambiguate ultrasound imaging and adapt to the neurosurgeon’s surgical objective, a patient-specific real-time network is trained using synthetic ultrasound data generated by simulating virtual iUS sweep acquisitions in pre-operative MR data. Extensive experiments performed in real ultrasound data demonstrate the effectiveness of the proposed approach, allowing for adapting to the surgeon’s definition of surgical targets and outperforming non-patient-specific models, neurosurgeon experts, and high-end tracking systems. Our code is available at: \url{https://github.com/ReubenDo/MHVAE-Seg}.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1788_paper.pdf
SharedIt Link: https://rdcu.be/dV5yl
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72089-5_45
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1788_supp.pdf
Link to the Code Repository
https://github.com/ReubenDo/MHVAE-Seg
Link to the Dataset(s)
https://doi.org/10.7937/3RAG-D070
BibTex
@InProceedings{Dor_PatientSpecific_MICCAI2024,
author = { Dorent, Reuben and Torio, Erickson and Haouchine, Nazim and Galvin, Colin and Frisken, Sarah and Golby, Alexandra and Kapur, Tina and Wells III, William M.},
title = { { Patient-Specific Real-Time Segmentation in Trackerless Brain Ultrasound } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15006},
month = {October},
page = {477 -- 487}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors claim that they developed a patient-specific real-time segmentation of brain in iUS. For that, the authors are combining pre-MRI and iUS for realizing the segmentation task. The method is trained using synthetic data and later validated in 7 cases of a public database.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The subject matter is pertinent and in line with the objectives of the conference. The authors assert that their work represents the first instance of segmenting freehand intraoperative ultrasound (iUS) leveraging pre-operative data. To achieve this, they propose generating realistic 2D iUS images from pre-MRI data, thus replicating the same anatomy with varying image appearances. The UNET model is trained using the synthetic database generated. Subsequently, the method is tested across seven cases; however, the absence of annual annotations in certain cases poses limitations on the method’s validation.
Of particular note is the computational time, which is remarkably low at 200 frames per second (FPS).
The main limitations of the technical contributions are presented in the conclusion, and were indicated as future work.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The current state-of-the-art remains relatively constrained, with several studies having already explored strategies that integrate MR+US for various tasks.
The testing solely on UNET is a limitation; exploring different architectures may be necessary to ensure optimal performance. The validation across seven cases is notably limited. While the strategy of training on synthetic data and testing on real data is common for Deep Learning models, the realism of synthetic data remains a concern. Validating the methodology requires demonstrating comparable performance when trained on synthetic versus real cases, which may be impractical due to the lack of available databases.
The authors should clarify the clinical practice regarding the necessity of pre-MRI before iUS and address how they handle potential variations between MRI acquisition and intervention.
In terms of technical advancements, this work does not introduce significant innovations but rather combines previously described methods.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The author must emphasize the clinical relevance of the described pipeline. Moreover, must indicate how these methods will advance the current clinical practice.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well aligned with the conference goal. Although no relevant technical are found, it is clear that this is a new application. The validation is interesting (although could be more completed) and shows the potential of the technique.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
Synthetic ultrasound from MRI is used to train a patient-specific tumor segmentation model for intra-operative cranial ultrasound, in order to allow for a real-time ultrasound segmentation that does not require a tracking / navigation system. Evaluation is performed on seven cases of a public MRI-US database.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This is an innovative idea and addresses a challenging problem. The overall paper presentation is also nice.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The computation time is very large, but model training can be launched once a pre-operative scan is available.
- The results are not entirely conclusive, because the different data sources and segmentation methods yield vastly varying accuracy (as you nicely illustrate in Figure 3 also). Even though you show promising results, it should be possible to eventually train a real-time iUS segmentation model that works accurately in a non-patient-specific manner, and is possibly even safer.
- If I am not mistaken, you are training and evaluating on the same database, but the only other 2D segmentation model comes from another one (ReMIND vs. RESECT), which is giving your method a favorable bias.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
It is nice work; apart from the comments above, I feel that the paper could be made a bit more clear. I had a hard time understanding which part of the evaluation is on 2D ultrasound frames, and 3D compounded volumes, and the multi-dimensionality of the evaluation (automatic / neurosurgeons; multiple data sources used) makes it hard to grasp the results quickly.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
See above.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper describes a method to segment brain tumors in intra-operative ultrasound images. The motivation behind the work is the potential of low-cost ultrasound imaging that is not fully exploited due to challenging interpretation. The paper describes a method to use synthetic US generated from pre-operative MRI with corresponding labels. The authors generate virtual US sweeps and uses MHVAE to synthesize US from MR slices. Finally, they train a segmentation network using data from k virtual sweeps from the same patient.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths
- Relevant and important problem
- Innovative use of data. The authors take advantage of the MR data with annotations which are much more accessible and easy to obtain than annotations of US.
- The results reach clinical relevance
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Navigation systems are widely used in neurosurgery and are essential for planning the craniotomy etc. The proposed method should complement the navigation system, not replace it. Tracked ultrasound is not necessarily a problem. It is more a question of avaliability in the commercial systems.
- Neuronavigation based on pre-op MR becomes invalid due to brain shift. Ultrasound based navigation does not become invalid and this is one of the main reasons to use US during surgery.
- For the simulation of sweeps: you need to know the positionning of the patient on the table to have a realistic trajectory. Not all openings are possible and for some tumor locations, several different approaches are possible. -The method requires a lot of input from the neurosurgeon regarding the procedure. Access to pre-op MR images, pre-op segmentations and planned craniotomy are needed. • The number of reference scans: The performance does not increase substantially, but the development duration becomes unacceptable for clinical use (3,5 hours for k=10). It seems unfeasible to perform such heavy computations during the (often) short time between planning and surgery.
- RESECT-Unet with 23 cases gives similar performance as BRATS subset with 611 cases. You avoid the US annotations, but the total annotation burden is still substantial
- Experiments on only 7 patients is very limited when you have a relatively large dataset available.
- Segmentation from navigation system: It is not entirely clear what this means. • Typo: aplying
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
The paper is largely based on open datasets, so the reproducibility should be good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- Comment on how this method should work together or as a complement to conventional neuronavigation.
- Explain how the method could work in a clinical setting with short deadlines and limited compute resources.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting work taking advantage of MR for US segmentation, but the setup seems to require too much input from the surgeon and too heay computation to be practical.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We thank the reviewers for their insightful comments. Reviewers found our work ‘pertinent’ and ‘new’ (R1), ‘innovative’ and ‘nice’ (R3), as well as ‘innovative,’ ‘relevant,’ and ‘interesting’ (R4). We are confident that minor edits can address most of the expressed concerns.
CLINICAL RELEVANCE (R1) Reviewer 1 mentioned, “The author must emphasize the clinical relevance of the described pipeline” and asked for clarification on “how these methods will advance current clinical practice.” We have elaborated on this in the revised manuscript, highlighting the potential impact of our framework on improving surgical accuracy and patient outcomes.
CLARIFICATION IN EVALUATION (R3) Reviewer 3 found it difficult to understand the evaluation, which is performed on 2D ultrasound frames but relies on segmentation performed in 3D pre-operative MRI with various annotation protocols. We have clarified this point in the revised version. First, we describe how 3D pre-operative segmentations with different protocols were obtained using 3D pre-operative MR scans. Then, we explain how ground truth annotations were derived for the 2D ultrasound images, via image registration.
We also would like to respectfully refute the claim that “it should be possible to eventually train a real-time iUS segmentation model that works accurately in a non-patient-specific manner and is possibly even safer.” The definition of the surgical target varies across neurosurgeons. Thus, we believe models should be patient-specific (and thus surgeon-specific). Moreover, ultrasound images are inherently ambiguous, and tissue boundaries are not always clearly visible. Our experiments demonstrate that general models and trained neurosurgeons familiar with iUS images struggle to identify tumor boundaries accurately. Therefore, in this work, we propose disambiguating ultrasound data by leveraging pre-operative data. As tumor boundaries in synthetic ultrasounds are not always clearly visible, our framework also relies on the geometry of the patient’s brain to better identify tumors.
ASSOCIATION WITH NEURONAVIGATION (R4) One of the motivations of this work is the limited integration of ultrasound imaging in current neuronavigation systems. This work aims to assist surgeons in better interpreting ultrasound images by automatically segmenting surgical targets based on pre-operative planning, without relying on neuronavigation systems. We agree that our framework could complement neuronavigation systems that support ultrasound imaging and have added this to the Future Work section: “From an application perspective, we will explore whether our framework can automatically detect significant misalignments between pre-operative and intra-operative data. This could alert surgeons to inaccuracies in the navigation system.”
COMPUTATION COST (R4) Our revised manuscript acknowledges the limitation regarding the computation cost of training a patient-specific model. However, as Reviewer 3 mentioned, “model training can be launched once a pre-operative scan is available.” We also respectfully refute the claim that it “requires too much input from the surgeon.” Firstly, our method does not require a “planned craniotomy” but simulates plausible craniotomies. Moreover, our method is fully automated and does not require any surgeon input if the BraTS protocol is used to identify the surgical target. If the surgeon defines the surgical target, input is needed. At our institution, this annotation process is already part of the routine clinical workflow, where surgeons annotate the surgical target the day before surgery for use in the neuronavigation system.
Meta-Review
Meta-review not available, early accepted paper.