List of Papers Browse by Subject Areas Author List
Abstract
Robotic ultrasound offers advantages over free-hand scanning, including improved reproducibility and reduced operator dependency. In clinical practice, US acquisition relies heavily on the sonographer’s experience and situational judgment. When transferring this process to robotic systems, such expertise is often encoded explicitly through fixed procedures and task-specific models, yielding pipelines that can be difficult to adapt to new scanning tasks. In this work, we propose a unified framework for autonomous robotic US scanning that leverages a large language model (LLM)–based agent to interpret US scanning guidelines and execute scans by dynamically invoking a set of provided software tools. Instead of encoding fixed scanning procedures, the LLM agent retrieves and reasons over guideline steps from scanning handbooks and adapts its planning decisions based on observations and the current scanning state. This enables the system to handle variable and decision-dependent workflows, such as adjusting scanning strategies, repeating steps, or selecting the appropriate next tool call in response to image quality or anatomical findings. Because the reasoning underlying tool selection is also critical for transparent and trustworthy planning, we further fine tune the LLM agent using a RL based strategy to improve both its reasoning quality and the correctness of tool selection and parameterization, while maintaining robust generalization to unseen guidelines and related tasks. We first validate the approach via verbal execution on 10 ultrasound scanning guidelines, assessing reasoning as well as tool selection and parameterization, and showing the benefit of RL fine tuning. We then demonstrate real world feasibility on robotic scanning of the gallbladder, spine, and kidney. Overall, the framework follows diverse guidelines and enables reliable autonomous scanning across multiple anatomical targets within a unified system.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2814_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/yuan-12138/RUSSAgent
Link to the Dataset(s)
N/A
BibTex
@InProceedings{BiYua_From_MICCAI2026,
author = { Bi, Yuan AND Zhou, Yiping AND Liu, Pei AND Li, Feng AND Jiang, Zhongliang AND Navab, Nassir},
title = { { From Scanning Guidelines to Action: A Robotic Ultrasound Agent with LLM-Based Reasoning } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16893},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper presents a unified and timely framework for autonomous robotic ultrasound scanning using an LLM-based agent. It translates scanning guidelines into dynamic tool use, integrating reasoning, planning, perception, and robot execution in a clinically relevant setting. Importantly, it achieves this without relying on heavily hand-crafted, task-specific pipelines, marking a meaningful step toward more general robotic ultrasound systems.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Novel and Compelling Framework: The idea of translating scanning guidelines into dynamic tool use via an LLM-based agent is highly compelling and timely. Unified System: The framework successfully integrates reasoning, planning, perception, and robot execution within a single agentic pipeline. Generalizability: The contribution is highly meaningful because it does not rely on heavily hand-crafted, task-specific pipelines, which is an important step toward more general robotic ultrasound systems. Clinical Feasibility: The real-world demonstrations on multiple anatomical targets (gallbladder, spine, and kidney) significantly strengthen the practical value and clinical relevance of the work.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Guideline Construction & Retrieval: It would be helpful to clarify how the scanning guidelines are constructed, formatted, and retrieved in practice. Since the guideline database is an important part of the framework, more detail on how these instructions are written and organized would improve reproducibility. RL Fine-tuning Details: The RL fine-tuning component is interesting, but the design choices in the reward function could be discussed in more depth. For example, what motivated the specific weighting terms in Eq. 1–4, and how sensitive is the final performance to these choices? Experimental Setup Clarity: For the verbal interaction experiments, it may be useful to further explain how the 20 guidelines were designed, how diverse they are, and to what extent the held-out guidelines differ from the training ones. Failure Mode Analysis: A brief discussion of failure modes—especially for the gallbladder case mentioned in the real-world experiments—would make the paper even stronger.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
There is also potential for more development in future directions. In particular, it would be interesting to know whether the proposed guideline-driven agent could be extended to additional anatomies, more complex interventional ultrasound settings, or broader human-robot collaborative imaging scenarios.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, I find this to be a strong systems paper, and potentially among the first works to formulate robotic ultrasound in this guideline-driven agentic manner. The overall paper is written clearly and the main idea is easy to follow. The real-world experiments effectively demonstrate the system’s capabilities. While there are places where the writing could be polished slightly for conciseness and clarity, and some methodological details (e.g., guideline construction, RL reward design) need further elaboration, the core contribution is solid and well-presented. Therefore, I lean towards acceptance.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes an agentic framework designed to achieve autonomous robotic ultrasound (US) scanning. The primary contribution over existing work is the reasoning capability that interprets system states and observations to guide robotic actions.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The introduction effectively contextualizes the problem, making the paper’s objectives and intended contributions easy to follow. The figures are well-designed, presenting the framework and pipeline clearly. The methodology pragmatically leverages existing software tools, suggesting a framework that could be highly effective in practice. The bridge between high-level LLM reasoning and low-level robotic control for specialized medical tasks like US scanning is highly relevant.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The authors claim “reasoning capability over state and observations” as a core contribution, yet the core method is unclear: (1) There is no formal definition of what constitutes a “state” or an “observation” within this framework. (2) It is unclear how raw sensory data and US images are converted into text-based tokens or descriptors that an LLM can process. Without this detail, the “reasoning” remains a “black box.” 2.The workflow for path generation and refinement lacks sufficient detail: (1) Localization of organs: While skeletal atlas registration is mentioned, the paper does not explain how soft-tissue organs (e.g., kidneys) are localized relative to the skeletal landmarks. (2) Path Definition: It is unclear if the scanning path is a simple linear interpolation between landmarks or another form that more sufficiently covers the anatomy. (3) Refinement & Segmentation: The trajectory refinement section lacks citations and technical specifics. Crucially, the authors do not specify which algorithm or model is utilized for the 3D US segmentation. 3.Experimental Validation: (1) While gallbladder scans include visual evidence, the results for spinal and kidney scans are presented only as text. Additional visual proof (e.g., reconstructed volumes or overlayed trajectories) or numerical results (e.g., success rates/reconstruction accuracy) will convince the reader better. (2) The paper does not report results from repeated trials. Without metrics on success rates or variance of trajectories across multiple runs, it is hard to assess the system’s robustness or statistical significance.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The integration of LLMs into medical robotics is a timely and promising direction. The methodology is also sound and can be effective in practice. However, the technical depth and experimental rigor of the manuscript can be improved. Specifically, the “reasoning” mechanism based on states and observations requires further description, and the real experimental validation can be more solid. Therefore, the final decision depends on the rebuttal.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
1.Developed a unified agentic system that uses LLMs to interpret clinical scanning handbooks via RAG, transforming static protocols into dynamic, decision-dependent robotic actions. 2.Introduced an RL-based fine-tuning strategy (using PPO) to improve the accuracy of tool selection, parameterization, and reasoning traces in smaller, computationally efficient models. 3.Integrated a complete suite of perception and control tools to successfully demonstrate autonomous scanning feasibility on human volunteers across the gallbladder, spine, and kidney.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
I appreciate the direction of this research, despite its imperfections. Utilizing an agentic approach for general-purpose reasoning to guide robotic motion offers a compelling solution for several scenarios. It effectively unleashes the reasoning potential of LLMs and provides a distinct advantage over other autonomous control schemes , even though its drawbacks are also clearly evident.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The reward function design for RL fine-tuning is overly focused on syntactic correctness, specifically tool naming and JSON schema matching, rather than clinical optimality. While this approach ensures the agent produces well-formatted and executable commands, it fails to bridge the gap between logical correctness and actual medical efficacy. Consequently, the model is incentivized to mimic reference traces rather than optimize for the best possible clinical outcome in a real-world setting.
- The architecture reduces complex, high-stakes medical maneuvers into a simplified sequence of textual reasoning and discrete tool calls. This paradigm may work for rigid, standardized protocols, but it lacks the high-frequency haptic feedback and sub-millisecond responsiveness required for safe physical interaction, making its true clinical utility in doubt. 3.Could the authors provide the average and peak latency for a single reasoning step (from observation input to tool call generation) on the hardware used? 4.How does the system handle instances where the patient’s breathing cycle is faster than the LLM’s reasoning frequency?
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method novelty and motivation
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers recognized the timeliness and relevance of our work, with R1 describing the idea of translating scanning guidelines into adaptive robotic scanning decisions via an LLM-based agent as “highly compelling and timely,” and R3 noting that the agentic approach offers a “compelling solution” with a “distinct advantage over other autonomous control schemes.” We also appreciate that R1 acknowledged the unified integration of reasoning, planning, perception, and robot execution, while R2 positively highlighted the clear problem contextualization, well-designed figures, and practical use of existing software tools. Importantly, R1 recognized the generalizability of our framework beyond hand-crafted, task-specific pipelines, and both R1 and R2 emphasized the relevance of bridging high-level LLM reasoning with low-level robotic control for robotic ultrasound. Finally, we thank R1 for recognizing that our real-world demonstrations on multiple anatomical targets strengthen the practical and clinical value of the work.
Our work proposes a guideline-driven agentic framework for autonomous robotic ultrasound, where the LLM serves as the high-level “brain” that retrieves scanning steps, reasons over the current state, and selects appropriate perception, planning, patient-guidance, and robot-execution tools. In contrast, the tools and low-level robot controllers act as the execution and reflex layers, handling real-time contact, force regulation, and robot motion. Therefore, the key contribution of this work is a unified hierarchical system that enables non-linear, decision-dependent scanning workflows beyond fixed rule-based pipelines, supported by RL fine-tuning and validated on held-out guidelines as well as real-world gallbladder, spine, and kidney scans
In this context, we would like to clarify the role of the LLM in our system (R3Q2 and R3Q4). The LLM is not intended for high-frequency physical control; instead, similar to human motor control, it performs high-level reasoning while dedicated low-level modules handle real-time contact, force regulation, and safety-critical interaction. Similarly, our RL fine-tuning is not designed to learn an unconstrained clinically optimal policy (R3Q1), but to make the LLM a reliable guideline-grounded planner. Given the dependence of robotic ultrasound on scanning guidelines and the risk of LLM hallucination, we intentionally prioritize faithful guideline following, correct tool selection, and valid parameterization, while recognizing patient-specific clinical optimization as an important future direction.
We also want to further clarify some minor implementation details regarding state representation (R2Q1) and path generation/refinement (R2Q2). Raw ultrasound and camera inputs are processed by the corresponding perception tools, whose outputs are returned to the LLM as structured text for high-level reasoning and tool selection. The current path generation is implemented as a simple trajectory connecting anatomical feature points. While more advanced path planning could be incorporated, this is not the main focus of the paper, and we will clarify this in the revision.
To further improve clarity and reproducibility, we will make several minor revisions following the reviewers’ suggestions. Specifically, we will briefly clarify guideline construction and the design of training/held-out guidelines, with code and guideline examples released for further details (R1Q1, R1Q3). We will also clarify the RL reward design (R1Q2), add a short failure-mode discussion for the gallbladder case (R1Q4), and strengthen experimental validation by adding repeated-trial numerical results and releasing an online video with additional visual evidence (R2Q3).
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All the reviewers recognize the proposed LLM-based agentic framework as both novel and timely, specifically the unified pipeline and the clinical feasibility demonstrated through real-world anatomical scans, including the gallbladder, spine, and kidney.
In summary, the proposed work presents a significant and meaningful step in medical ultrasound robotics, and I lean towards early acceptance, However, authors should address the limitations raised by the reviewers, e.g. clarity of state/observation definition and sensor data to text conversion. More details are needed on guideline construction, segmentation algorithms and RL reward function, and consider including metrics such as success rates and latency.
