Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for surgical instrumentation that combines multi-camera markerfree 6D pose estimation with augmented reality (AR) in-situ visualization on a head-mounted display. Pose estimation and consecutive camera calibration are achieved through known objects. Our 6D pose estimation is purely trained on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch assembly steps in intraoperative use.

In a technical evaluation, we outperform state-of-art 6D pose estimation. A user study with 29 scrub nurses conducted in a surgical simulation of knee arthroplasty compares our system against a paper manual. Our AR system significantly reduced the perceived workload. Objectively, AR guidance reduced task completion time by 21.3% (4.76 minutes). Specifically, less-experienced scrub nurses benefited from our approach. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0941_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: https://papers.miccai.org/miccai-2026/supp/0941_supp.zip

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{LiShi_Multicamera_MICCAI2026,
        author = { Li, Shiyu AND Kreimeier, Julian AND Schieber, Hannah AND Müller, Dirk AND Kainz, Bernhard AND von Eisenhart-Rothe, Rüdiger AND Roth, Daniel},
        title = { { Multi-camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16893},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    Addressing a to-date relatively underexplored research area of AR-assisted surgical instrument handling, the authors present a multi-camera guidance system designed to support scrub nurses during assistive tasks in knee arthroplasty. The proposed AR guidance system leverages AI-assisted computer vision to detect surgical trays and an assembly table containing mechanical components required for knee arthroplasty. Once this equipment is localized, AR-based visual guidance is provided to the scrub nurse through tooltips and assembly animations, rendered as 3D virtual objects and displayed via a Microsoft HoloLens 2 optical see-through head-mounted display (OST-HMD). More specifically, the authors claim four main contributions: 1.) An AR guidance system tailored for surgical instrument handling and assembly, utilizing deep learning-based 6-degree-of-freedom (6DoF) pose estimation of surgical tools. 2.) A multi-camera pose estimation approach employing four Azure Kinect cameras in combination with a HoloLens 2, trained exclusively on synthetic data to bypass annotation challenges and facilitate generalization to unseen surgical instruments. 3.) A technical evaluation of the proposed system, including runtime performance and pose estimation accuracy. 4.) A user study in which scrub nurses compare the AR-assisted system against a conventional gold-standard approach.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.) The paper provides a strong four-fold contribution, including a certain degree of domain-specific technical novelty and a user study. Sufficient novelty is demonstrated by the proposed application context (pick, assemble, and handover of surgical instruments by scrub nurses) and the multi-camera deep learning-based pose estimation method, particularly as it is trained purely on synthetic data. Moreover, benchmarking the proposed multi-camera AR assembly guidance within the given surgical context opens up opportunities for future research contributions in this domain. 2.) A valuable novel method demonstrating the potential of OST-HMD-based AR to support the training of inexperienced users in a medical context. 3.) High-quality supplementary material, including videos and an additional document detailing the multi-camera pose estimation method. 4.) The paper addresses a relatively underexplored aspect of AR in knee arthroplasty, namely intraoperative instrument handling and assembly.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.) The domain gap between synthetic and real data is not sufficiently addressed. The authors appear somewhat optimistic about the effectiveness of a purely synthetic data-driven approach. 2.) While the use of synthetic data offers clear advantages, the authors should also discuss the potential inclusion of real annotated data for future performance comparisons of the deep learning models. 3.) The Microsoft HoloLens 2 is a discontinued OST-HMD, which may limit the long-term applicability and future-proofness of the proposed system. This aspect should be acknowledged and discussed in the manuscript. Table 2 would benefit from a legend explaining the NASA-TLX-specific column headers (P_M, P_SD, ARM_M, AR_SD), as these may not be immediately intuitive to all readers. 4.) Reproducibility is somewhat limited due to the absence of publicly available source code. While the workflow is generally well described, certain steps would benefit from additional detail. For example, the artifact mitigation process using MeshLab is not sufficiently specified (e. g. , which exact processing steps were applied). Similarly, the domain randomization process implemented with BlenderProc lacks detail. It would be helpful to specify the backgrounds, lighting conditions, camera poses, and distractor objects used during training.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Dear authors, I consider this contribution very valuable; however, it could be improved by a few minor revisions, as detailed under “Main weaknesses of the contribution. “ I hope the suggestions mentioned there are helpful.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall, it appears to be a solid piece of work that should be published after some minor revisions.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper presents a multi-camera, marker-free AR guidance system for surgical instrument handling and assembly. The system integrates 6D pose estimation with in-situ AR visualization to provide step-by-step guidance for scrub nurses. A user study with 29 participants demonstrates reduced task completion time and perceived workload compared to standard paper-based manuals.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses an underexplored problem in the OR workflow. The motivation is well grounded in real clinical challenges, including cognitive load, unfamiliar instruments, and workflow interruptions. 2.The work presents a comprehensive end-to-end system that integrates multi-camera pose estimation, AR visualization, and interaction design, demonstrating a substantial engineering effort. 3.The system is further validated through a user study with 29 healthcare professionals, which is a strong aspect of the paper. The study shows statistically improvements in task completion time and perceived workload, supporting the practical effectiveness of the approach.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The technical novelty is limited, as the proposed system mainly integrates existing components for detection, pose estimation, multi-camera fusion, and AR guidance, rather than introducing a substantially new algorithmic contribution. 2.The multi-camera calibration is implicitly achieved via object-based pose alignment using known instruments. The calibration accuracy depends heavily on the quality of object pose estimation. The paper does not provide an evaluation of calibration accuracy or comparisons, making it difficult to assess its reliability.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses a clinically relevant and underexplored problem in the OR workflow and presents a comprehensive AR guidance system for surgical instrument handling and assembly. A major strength of the work is the user study with 29 healthcare professionals, which shows statistically significant improvements in task completion time and perceived workload. The system integration effort is substantial, combining multi-camera pose estimation, AR visualization, and user interaction in a surgical setting. Overall, I find the paper to be a practical application paper with a strong user study, but with only moderate methodological novelty.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper introduces an AR system to assist nurses in identifying and using the correct surgical tools. To achieve this, a multi-camera markerless object pose tracking method is used, though I believe this is mostly based on a prior paper (?). A nice user study shows that the system helps more novice nurses.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is interesting, relevant, and generally well written. The system appears technically sound, and the user study has a good sample size and promising results.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Below are some specific comments, but my overall concern is that the clarity is lacking in parts, making it difficult to understand exactly what was done. For example, it is unclear to what degree the markerless pose tracking method is new in this paper or uses prior work.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • First sentence of Introduction: “environment” has a typo. Also do the authors mean sensitive, not sensible?
    • Introduction, second sentence: the authors mention endoprosthetic instruments, but there is no other explanation of endoprosthetic surgery. Are the presented techniques specific to that? It seems like they don’t have to be limited thus?
    • On page 2, first sentence, what do the authors mean by “this is not currently the gold standard”? Does gold standard treatment exceed or fail to meet this requirement? Or is the implication that fulfilling this requirement still doesn’t guarantee gold standard level treatment because they just use paper manuals?
    • Is the proposed pose estimation system different from the one in ref [3]? In general, it’s not clear if the pose tracking is new in this paper, or based on prior work.
    • How is occlusion handled, for example by the nurse’s head if he/she is leaning over the table, or by his/her hand?
    • Some of the acronyms should be defined (DDC, RTMO, ADD(-S), etc)
    • The TCP streaming from PC to HMD only sends the computed poses over a local wifi network, correct? Or is the HMD video streamed to the inference PC as well? If so, is an encoding used?
    • Section 2.3 mentions a 200ms runtime and a 3s runtime, but it’s not completely clear what each refers to. In the video it seems like the 3 seconds is to update the tool poses. Why is it so slow? Is that just the inference time of the vision model? It seems to conflict with the runtimes in table 1.- It should be more clearly explained how the HMD is registered to the static cameras and what the accuracy of this registration is.
    • How is this methodology scalable to support all required instruments? Would it involve retraining the model with every new instrument in turn? The Discussion mentions manufacturer CAD models, but in my experience these are often not available or not photorealistic.
    • In Section 3 and Table 1, it is not clear what is meant by camera pose. Also, the translation values seem rather large. is 6-7cm acceptable?
    • Just below Table 1, it says “We recruited 29 scrub nurses () participated.” The last word can be removed.
    • Fig 4 mid is interesting and it is tempting to conclude that the guidance is useful for nurses with <4 years experience. However, most of the usage counts have n=1, so this is quite possibly chance. The conclusions from this should probably be softened.
    • Related to this, in the figure caption it says 4 uses, but in the text below it says 6 uses
    • Table 2 should include explanation of the acronyms in the caption (P_M, P_SD, AR_M, AR_SD)
    • Why did the authors use the HoloLens 2 if it has been discontinued for a few years now?
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall a worthy contribution and good user study. Missing clarity and details brought my score down a bit.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

Dear coordinator and reviewers,

We would like to express our gratitude for your time and patience and thank you for the detailed feedback. We will submit a camera-ready version for our provisionally accepted paper. In the following, we provide detailed information on how we address the remaining comments and incorporate the reviewers’ feedback.

Introduction and related work Regarding the motivation for selecting endoprosthetic surgery (R3), we better clarified that it represents a typical and complex open-surgery use case. Regarding the use of the paper manual as baseline material (R3), we also clarified that it represents the commonly used reference material for surgical instrument guidance and is therefore used as the baseline comparison in our study.

Contribution and technical novelty Regarding the technical novelty and distinction from prior work (R2, R3), we refined the discussion of the hierarchical pose estimation framework for instruments and trays, including multi-view fusion and filtering methods tailored to the surgical setting. We further clarified that the work focuses on adapting and extending pose estimation methods for AR-guided surgical workflow support, particularly for inexperienced scrub nurses.

Technical details Regarding the domain gap between synthetic and real data (R1), we use synthetic-data generation and domain randomization, including pose and lighting variation strategies used to mitigate the synthetic-to-real domain gap. We better clarified the reasoning and that plausibility filtering is used to suppress implausible or unstable pose estimates. Regarding implementation details (R1, R3), we added further information on MeshLab postprocessing and the TCP-based communication pipeline between the PC and HoloLens 2.Regarding the choice of HoloLens 2 (R1, R3), we further justified the use of an optical see-through HMD for surgical environments. Regarding latency and occlusions (R3), we clarified that the latency reported in Tab. 1 refers to runtime latency, independent of the HMD visualization update frequency. We also clarified that the evaluated setup primarily uses top-down camera views with limited major occlusions. During temporary hand occlusions, objects may not be detected; in such cases, the filtering approach maintains the last reliable pose estimate for static tray objects and for objects outside the trays. Regarding reproducibility (R1, R3), we improved reproducibility by adding further technical details and including the now non-anonymized reference.

Evaluation Regarding evaluation metrics and annotations (R1, R2, R3), we clarified the metrics in Tabs. 1 and 2, including ADD(-S) and NASA-TLX terminology, by adding explanatory notes and improving table descriptions. We made the conclusion on the turnover point of usage more precise (R3). Regarding camera pose evaluation (R2, R3), we already reported the accuracy of camera extrinsics in Table 1 and clarified that the reported values are averaged across all evaluation conditions.

Limitations Regarding scalability (R3), we included that broader generalization to unseen tools and procedures remains future work.

We corrected spelling mistakes and specified the discussion in the paper.

Thank you again for your valuable feedback.

Sincerely, The authors



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper introduces an AR system to assist the scrub nurse during knee arthroplasty. The system comprises a multi-camera system that detects, tracks, and estimates the 6D pose of marker-free trays and instruments, and an HDM that is worn by the nurse for step-by-step guidance on picking, assembling and handling those trays and instruments. The system is evaluated in a user study with 29 subjects that demonstrate it helps novice nurses. The reviewer agrees that, although technical novelty is moderate, the paper addresses a clinically relevant, unexplored problem in the OR, with the system being a first of the kind. It is an interesting practical application paper that received an unanimous recommendation of acceptance that I support. I encourage the authors to incorporate as much as possible the received feedback (improve clarity, accuracy of camera extrinsic calibration, etc)



back to top