List of Papers Browse by Subject Areas Author List
Abstract
We present in this paper a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. This disentanglement is achieved by controlling a Neural Radiance Field’s appearance with a multi-style hypernetwork. Once trained, the implicit neural representation serves as a differentiable rendering engine, which can be used to estimate the surgical camera pose by minimizing the dissimilarity between its rendered images and the target intraoperative image. We tested our method on retrospective patients’ data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/3581_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: N/A
Link to the Code Repository
https://maxfehrentz.github.io/style-ngp/
Link to the Dataset(s)
https://maxfehrentz.github.io/style-ngp/
BibTex
@InProceedings{Feh_Intraoperative_MICCAI2024,
author = { Fehrentz, Maximilian and Azampour, Mohammad Farid and Dorent, Reuben and Rasheed, Hassan and Galvin, Colin and Golby, Alexandra and Wells III, William M. and Frisken, Sarah and Navab, Nassir and Haouchine, Nazim},
title = { { Intraoperative Registration by Cross-Modal Inverse Neural Rendering } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15006},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces a 3D/2D intraoperative registration method for neurosurgery, employing cross-modal inverse neural rendering. The experimental evaluation assesses the potential of the approach to align preoperative Magnetic Resonance (MR) images with intraoperative surgical views of the brain surface obtained post-craniotomy and captured using a camera. Both synthetic and retrospective clinical data are utilized.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper introduces the application of Neural Radiance Fields for multimodal 3D/2D registration. It proposes a novel formulation that separates Neural Radiance Fields into structural and appearance representations. The anatomical structure is learned preoperatively and appearance is adapted intraoperatively. The use of a hypernetwork to adjust the appearance of Neural Radiance Fields intraoperatively is particularly noteworthy. It represents an innovative approach to extend the use of Neural Radiance Fields for multimodal registration where appearances differ from target images, as in intraoperative scenarios. Application of the method on clinical cases demonstrates its efficacy in aligning preoperative MR images with intraoperative surgical views post-craniotomy.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The evaluation is limited and heavily reliant on visual assessment. The paper overlooks the challenges of 3D/2D registration in scenes with occlusion by surgical tools.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The paper’s reproducibility is limited by the absence of publicly available code, and the unavailability of a public dataset.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- Quantitative metrics are necessary to assess the accuracy of the hypernetwork in generating plausible appearances and to evaluate registration in clinical cases.
- Given the computationally intensive nature of the NeRF algorithm, with processing times ranging from hours to days for complex scenes, what are the computation times for each step (pre- and intraoperative) in the image registration process?
- How is the 3D mesh M, employed to characterize anatomical structure, obtained? What level of precision is required for the mesh, and how might it impact the reliability of the final image registration?
- Please provide clarification on the typical dimensions and voxel sizes of the images used in this study.
- Could the authors confirm whether the five cases represent distinct patients?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Nice work, addressing an important and very challenging clinical problem.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper introduces an approach for 3D/2D intraoperative registration during neurosurgery using cross-modal inverse neural rendering. The core innovation lies in the disentanglement of a Neural Radiance Field into structural and appearance components, controlled by a hypernetwork that adjusts appearance intraoperatively while maintaining the preoperative anatomical structure. This method facilitates surgical camera pose estimation through a differentiable rendering engine, which aligns rendered images with intraoperative imaging.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper presents several notable strengths:
Novel Formulation: The proposed method innovatively separates the Neural Radiance Field (NeRF) into anatomical structure and appearance components, where the appearance is dynamically adapted during surgery using a multi-style hypernetwork. This separation allows for accurate adaptation to intraoperative conditions without altering the anatomical accuracy learned from preoperative data. Original Use of Data: The application of a hypernetwork to control NeRF’s appearance component based on intraoperative imagery is a highly original utilization of available data. This approach effectively bridges the modality gap between preoperative and intraoperative environments. Strong Evaluation: The evaluation conducted includes both synthetic and real patient data, demonstrating the method’s superiority over existing state-of-the-art techniques. The rigorous testing under clinical standards ensures the method’s applicability in real-world settings. Clinical Feasibility: The technique eliminates the need for additional imaging acquisitions or cumbersome optical tracking systems, integrating seamlessly with existing surgical workflows and technology. This feasibility is crucial for practical deployment in neurosurgical settings. Novel Application in a Clinical Setting: Applying NeRF and hypernetworks for intraoperative registration in neurosurgery showcases a pioneering step in medical imaging, potentially enhancing surgical precision and patient outcomes.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
However, there are some areas where the paper could be improved: Lack of Comparison with a Broader Range of Existing Methods: While the paper compares the proposed method against specific state-of-the-art techniques, it lacks a broader analysis against a wider range of existing methods, which could include other advanced machine learning or computer vision approaches used in similar registration tasks. Potential Overfitting on Style Adaptation: The method’s heavy reliance on a hypernetwork for style adaptation could lead to overfitting, especially with limited intraoperative data available during actual procedures. This risk is not adequately addressed, and the generalizability of the hypernetwork across diverse surgical scenarios remains questionable. Complexity in Implementation: The technical complexity of implementing such an advanced system involving NeRFs and hypernetworks might limit its adoption in less technologically advanced medical facilities. The operational requirements and computational costs associated with the proposed method are not thoroughly discussed. Limited Discussion on Failure Cases: The paper briefly mentions cases where the method failed (e.g., early local minimum in optimization) but does not provide a detailed analysis or potential solutions to these issues, which are critical for clinical applications.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
n/a
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
To enhance the paper, consider the following suggestions: Broaden Comparative Analysis: Expand the range of comparative evaluations to include more diverse methodologies. This broader comparison would help in highlighting the unique benefits and potential limitations of the proposed approach more clearly. Address Overfitting Concerns: Provide a deeper analysis of how the hypernetwork handles intraoperative variability and discuss strategies to prevent overfitting. Including regularization techniques or expanding the training dataset might be beneficial. Detail Implementation Challenges: Offer more details on the computational requirements and implementation logistics. Discuss how the system could be integrated into existing surgical infrastructures and any required modifications. Enhance Discussion on Limitations and Failures: Elaborate on the scenarios where the method did not perform as expected. Analyzing these cases will provide valuable insights into the system’s robustness and areas for further improvement. Future Work: Clearly outline potential areas for future research, such as extending the method to handle non-rigid deformations or exploring more efficient and scalable training techniques for the hypernetwork.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The decision to recommend this paper for acceptance is based on its technical innovation, evaluation metrics, and its practical applicability in a clinical setting. The novel approach of using a hypernetwork to dynamically adapt the appearance of a NeRF during surgery represents a substantial advancement in the field of medical imaging and surgical navigation. Despite some limitations, such as the potential for overfitting and the high technical complexity, the benefits in terms of increased surgical accuracy and reduced operational burden significantly outweigh these issues. The thorough testing and clear clinical relevance further support the paper’s high value to the community, making it a strong candidate for acceptance at the conference.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The paper introduces a novel approach for neurosurgical registration using implicit neural representations. This method separates the structural and appearance representations of the neural radiance fields (NeRFs), allowing adaptation of appearance intraoperatively while maintaining the learned anatomical structure. The authors test the proposed pipeline on both synthetic and real data to evaluate the effectiveness of the proposed approach.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea of utilizing NeRF to first learn anatomical information and then learn single-shot style adapatation for neurosurgical registration with cross-modality data is interesting.
- The authors provide a reasonable solution to the proposed problem, based on the evaluation results presented in the paper.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I am a bit concerned about the scale of the evaluation dataset. If I understand correctly, in Section 3, the authors decscribed the dataset as including only 5 clinical cases only, which is relatively small. On the other hand, the description of the training of the mesh stylization is also a bit confusing, do you use the images from 4 out of 5 testing cases to train the network? If that’s the case, how the proposed pipeline can be generalized/adapted in real-world scenarios where the network won’t be able to access the real clinical images before deployment?
- It would be interesting if the authors could discuss the generalizability of the proposed method. It seems that the proposed pipeline heavily relies on the features on the vessels from the brain surfaces for registration, could the proposed pipeline potentially be applicable to the cases where the craniotomy is not performed, but only face RGBD images are accessible for registration?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Please refer to Section 5 and 6 for further details.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method does provide a feasible solution to the intraoperative registration problem, with certain limitations/flaws (please refer to Section 6 for further details), thus I’d like to check the authors’ response in rebuttal before finalizing my rating.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We are pleased that the reviewers recognized the novelty and technical strength of our work, particularly highlighting our innovative approach to using Neural Radiance Fields in conjunction with a hypernetwork for multimodal 3D/2D registration in neurosurgery. R1 noted the novel formulation and application of NeRF, along with its clinical feasibility, while R3 appreciated the rigorous evaluation on clinical data and the method’s potential to be used in clinical practice to enhance surgical precision. R4 also acknowledged the originality of our separation of learning anatomy and appearance. We value the constructive feedback provided and address the major points of critique below.
EVALUATION AND COMPARISON: The reviewers pointed out the need for a more rigorous evaluation of the hypernetwork and a more extensive (quantitative) comparison with a broader range of existing methods. Our main goal for the hypernetwork was to show that it can be effectively used - even on little data - to go from MR to RGB appearance with sufficient similarity to the intraoperative target to enable registration with conventional loss functions. To show that, we focused on evaluating the downstream task rather than the synthesis quality of the hypernet, as an extensive exploration of features, architecture, and interface of hypernets with radiance fields was not the primary focus of this paper. Our evaluation for the clinical cases with corresponding target images was qualitative since manually labeled 6 DoF ground truths are difficult to acquire and come with low inter-annotator agreement. Instead, we opted for an extensive quantitative evaluation on synthetic targets with accurate ground truth poses and provided rotation and translation errors. For comparison, we limited ourselves to SOTA methodologies that come with the same simple prerequisites (single RGB image from the surgical microscope) as our method. Other SOTA methods (such as skull-based ICP) require additional data acquisition in the OR and were therefore not considered.
GENERALIZABILITY AND OVERFITTING (R3, R4): Whereas the reliance on vessel features is an inherent limitation of our approach that we will clarify, the danger of overfitting the hypernet on limited data can be solved in future work with 1) more elaborate data synthesis and 2) additional data acquisition. We refrained from employing 1) since a larger synthetic dataset would still not have guaranteed that we capture the “real” distribution of brain surface appearances. For 2), data acquisition is ongoing.
FAILURE CASES: While the clinical feasibility of our method was acknowledged, reviewers suggested providing a more detailed discussion on failure cases. In an additional future work section, we will elaborate on how we can make our method more robust and propose solutions to failure cases.
COMPUTATIONAL REQUIREMENTS: Reviewers raised concerns regarding the computational intensity. Since all training is done preoperatively, the computational complexity is heavily concentrated in that phase and has a limited impact on intraoperative registration. We will add details on the computational requirements in the final version.
FUTURE WORK: The reviewers pointed out that there is no dedicated future work section. We will elaborate on this in the final version. This will cover potential extensions of our method to handle non-rigid deformations, exploration of the hypernet, and a better pose solver that can go beyond pixel-wise loss and instead operates on the whole image to allow for more robust loss functions.
We will add technical details on data dimensionality, scalability, and data acquisition process in the final paper. The source code, documentation, and a dataset will be made public.
Meta-Review
Meta-review not available, early accepted paper.