Abstract

Vision-Language-Action (VLA) models represent a promising direction for embodied intelligence in surgical robotics. Despite the prevalence of VLA benchmarks for general robotics, standardized evaluation platforms specifically designed for surgical contexts remain absent. To address this limitation, we present SurgVLA-Bench, the first comprehensive benchmark for evaluating VLA models in laparoscopic surgical robotics. Leveraging the SurRoL simulation platform, we construct a hierarchical task taxonomy ranging from atomic actions to complete surgical procedures, complemented by a multi-dimensional evaluation framework assessing action accuracy and semantic consistency. We then systematically evaluate two representative paradigms, including autoregressive models such as OpenVLA, and flow matching models such as $\pi_{0}$, $\pi_{0.5}$, and SmolVLA. Our experiments show that autoregressive models tend to excel in semantic understanding, while flow matching models often achieve higher task precision but may face generalization trade-offs. However, even the best-performing models remain far from satisfactory, as the constrained endoscopic field of view, restricted viewing angles, and frequent occlusions persist as fundamental physical bottlenecks. The code and data are available at https://github.com/VCL-HNU/SurgVLA .

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1457_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/VCL-HNU/SurgVLA

Link to the Dataset(s)

https://huggingface.co/datasets/Kanden1112/surg-vla-dataset

BibTex

@InProceedings{SunJia_SurgVLABench_MICCAI2026,
        author = { Sun, Jiashuo AND He, Yue AND Liu, Wenxuan AND Mao, Tao AND Wang, Jiazheng AND Chen, Xiang AND Liu, Min},
        title = { { SurgVLA-Bench: Towards Evaluating Vision-Language-Action Models for Laparoscopic Surgical Robotics } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16895},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This work proposes SurgVLA-bench, a benchmark framework for surgical VLA models. The framework facilitates the synthesis of the dataset necessary for training VLA models and allows for the validation of trained models

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    With the emergence of VLA models and the timely attention towards automation of robot actions, this framework, if observed, has results similar to real-world, and offers a flexible platform for rapid VLA development and testing.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The domain gap between simulation (in the framework) and the real world seems big. 2.The ability/flexibility of the framework to generate similar (allowing the model to overfit for a task) and diverse clinically relevant data (allowing the model to generalize based on diverse data) is unclear 3.The actual relation between the model’s success in simulation and the real world is undetermined. 4.The realism of deformable objects/tissue is unknown

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This work has great potential and could play a key role in design and validation of models for surgical automation. However, in its current state, it is unclear to what extent it can realistically replicate real-world scenes and challenges, and if there is any correlation between model performance in this framework and real-world has clincial relavance.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    In my view, this work has limited extension from prior work (Surrol). Strong motivations and problem formulation (benchmarking platform) are not sufficiently justified by experiments/results/demo application. With this platform being pitched as a validator for VLAs, in my view, it is not sufficiently proven. For instance, VLAs are sensitive to visual domain shift and soft tissue interaction. So, the likelihood of a VLA model (that is successful in this platform) succeeding in a real tissue interaction is unclear (and highly unlikely).



Review #2

  • Please describe the contribution of the paper

    This paper introduces SurgVLA-Bench, a benchmark for evaluating Vision-Language-Action (VLA) models in laparoscopic surgical robotics. Built on the SurRoL simulation platform, the benchmark defines a hierarchical task taxonomy spanning three difficulty levels and eight tasks ( from atomic actions to the hemostatic procedure involving sequential steps). The paper constructs a dataset of ~800 trajectories and evaluate four representative VLA models (OpenVLA, pi0, pi0.5, SmolVLA).

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Hierarchical task design: The three-level taxonomy is well-structured, with each level targeting distinct capability dimensions.
    • Model coverage: Evaluating autoregressive and flow matching paradigms provides insights about trade-offs between semantic understanding and action precision.
    • Beyond success rate analysis: Additional experiments on prompt robustness and cross-level generalization provide useful insights about the limitations of state-of-the-art models in the surgical domain.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The paper’s primary contribution is the benchmark itself rather than a novel method. The dataset construction, simulation environment, and evaluation protocol rely on existing platforms (SurRoL, LoRA fine-tuning) without architectural or methodological innovation. This is acceptable for a benchmarking paper, but the paper should clarify what the expected technical contribution of future work building on SurgVLA-Bench would be.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Despite limited technical novelty, the benchmark is a useful resource for VLAs in surgical environments with a well-designed evaluation framework.

  • Reviewer confidence

    Not confident (1)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The paper presents an integrated vision-language-action framework in surgery in a simulated environment. Even though there is a clear sim-to-real gap, it is a useful resource for early surgical VLA benchmarking in a practical and reproducible manner, allowing the community to advance in the field and compare new methods against available baselines with the hierarchical tasks and metrics proposed in this work. Additionally, the benchmarking study and analysis of existing general-domain VLAs adapted to this scenario is valuable to understand strengths and weaknesses of existing models regarding transferability. Therefore, my recommendation is to accept this paper as it represents a contribution to the surgical data science community.



Review #3

  • Please describe the contribution of the paper

    This paper presents SurgVLA-Bench, the first benchmark specifically designed for evaluating Vision-Language-Action models in surgical robotics. Built upon the SurRoL simulation platform, it introduces a hierarchical evaluation framework spanning three difficulty levels and eight tasks, covering a wide range of capabilities from atomic actions to complete surgical procedures. The authors also construct a standardized surgical VLA dataset, provided in multiple formats including LeRobot, RLDS, and 3D point cloud, facilitating direct adoption and future extension by the research community. Four representative VLA models are evaluated across several dimensions including success rate, prompt robustness, and cross-task generalization. Experimental results reveal that autoregressive models tend to excel at semantic understanding, while flow matching models demonstrate superior precision in task execution, yet none of the evaluated models achieve satisfactory performance under multi-task training and testing conditions.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Hierarchical Evaluation Framework. The paper introduces a three-level hierarchical benchmark that progressively increases in complexity, ranging from atomic actions to multi-step surgical procedures. This design enables fine-grained assessment of surgical VLA models at different difficulty stages, providing richer diagnostic insights than a single overall performance metric.

    2.Dedicated Dataset with Extensibility and Multi-format Support. The authors construct a surgical task dataset from scratch, collecting over 800 manually verified trajectories across eight tasks, addressing the lack of surgical data for VLA research. The dataset will be released in three mainstream formats (LeRobot, RLDS, and 3D point cloud), and an integrated data collection pipeline is provided to support future dataset scaling.

    3.Evaluation of Representative VLA Models. Four representative VLA models covering two mainstream paradigms are evaluated not only on success rate, but also across prompt robustness, instruction informativeness utilization, and cross-task generalization.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Lack of Zero-shot Baseline. In the introduction, the authors claim that VLA models trained on general domains lack applicability to surgical contexts. However, in the experimental section, all four VLA models are evaluated only after LoRA fine-tuning on the SurgVLA-Bench dataset. No zero-shot performance is reported as a baseline, leaving the claim insufficiently supported by quantitative evidence.

    2.Inconsistency in SmolVLA Performance Reporting. In Table 2, SmolVLA achieves a success rate (SR) of 0% on task T1-1.However, in the Analysis of Success Rate section, the authors report that SmolVLA achieves 82% SR on T1-1 under single-task training. Please clarify this inconsistency.

    3.Insufficient Explanation for Generalization Results In the Analysis of Generalization section, the authors attribute OpenVLA’s notable performance gain on T1-1 after transfer to the cleaner training signal in Level 1, where interference from other objects is absent. However, this explanation is unconvincing, since T2-1 (the corresponding gauze grasping task in Level 2) also involves only the gauze object without distractor objects, yet a slight performance degradation is observed after transfer. This contradicts the proposed explanation.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    1.Insufficient Baseline Comparisons. Without zero-shot performance as a reference, it is difficult to determine whether the observed low performance stems from the inherent difficulty of the surgical tasks or from the intrinsic limitations of the selected models, leaving the central claim that general-domain VLA models lack applicability to surgical contexts insufficiently supported.

    2.Inconsistency Between Reported Results and Analysis. There are discrepancies between the numerical results presented in the tables and the descriptions provided in the analysis section. The inconsistency undermines the reliability of the reported findings.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors addressed all my concerns. Overall, the benchmark proposed in this paper makes a meaningful contribution to the surgical VLA community.



Author Feedback

We sincerely thank the reviewers for their constructive feedback. We address each concern below. Reviewer 1: Q1&3: Domain Gap and Sim-to-Real Feasibility R1&3: We acknowledge the visual gap between simulation and clinical settings. However: 1)Prior work[1] has demonstrated successful sim-to-real transfer for this setup, confirming the practical relevance of our findings. Due to the difficulty of obtaining consistent real-world surgical data, simulation platforms such as SurRoL are increasingly serving as a key infrastructure for benchmarking surgical models. 2) Our simulation replicates real-world 7-DOF robotic kinematics and task success criteria, which are the decisive factors for task performance. 3) The visual gap can be effectively mitigated via style transfer.

[1] Long Y, et al. Surgical embodied intelligence for generalized task autonomy in laparoscopic robot-assisted surgery. Science Robotics, 2025.Q2: Framework Flexibility and Data Diversity R2: SurgVLA-Bench covers atomic operations (grasp, place, reach) that form the basis of most surgical procedures. The framework’s modular design allows researchers to easily integrate new tasks by replacing anatomical models or adjusting trajectories. This capability to efficiently generate diverse datasets addresses the dual evaluation needs of assessing both task-specific proficiency and cross-scenario generalization. Q4: Realism of Deformable Tissue R4: We acknowledge certain surgical tasks, such as retraction and dissection, inherently require realistic tissue deformation. However, our current tasks can be completed without tissue deformation, so the evaluation remains valid. Realistic modeling of tissue deformation for complex maneuvers is a key focus of our future work.

Reviewer 2: Q1: Platform Innovation and Future Work R1: Our work focuses on benchmarking rather than VLA architectural innovation. Our core contributions are: 1) We provide the first open-source (after acceptance), reproducible, multi-level evaluation platform for surgical VLA models, establishing a standardized pipeline that the community currently lacks. 2) The multi-layered task design systematically uncovers capability bottlenecks that single-task setups cannot reveal, providing clear directions for future surgical VLA research. Based on these contributions, we will expand the framework with more diverse tasks and multi-dimensional metrics to offer a comprehensive baseline for comparison.

Reviewer 4: Q1: Lack of Zero-shot Baseline R1: Initial zero-shot tests yielded 0% success across all tasks, so we did not report them as a separate baseline. The reasons are: 1)General VLAs lack exposure to surgical instruments and anatomy. 2)Surgical kinematics differ from industrial settings. Therefore, zero-shot evaluation is not necessary, and we performed domain-specific fine-tuning and then conducted comparison. A full explanation and analysis will be included in the final version. Q2: Inconsistency in SmolVLA Performance Reporting R2: The 0% success rate in Table 2 comes from a multi-task training setup where task interference occurs. The 82% success rate mentioned in the text is from a single-task training setup. This stark gap shows that lightweight models like SmolVLA suffer from severe interference during multi-task learning, highlighting the need for more robust multi-task architectures in surgical scenarios. Q3: Insufficient Explanation for Generalization Results R3: T2-1 is not a distractor-free environment; it is a multi-object scene where instruments such as needles act as deliberate distractors. The performance drop compared to distractor-free T1-1 is therefore expected. Architecturally, unlike models such as π0.5 that employ explicit vision-language alignment, OpenVLA fuses image and language embeddings directly within the LLM. This leads to biases in understanding similar language tokens and a tendency to overfit to the scene, impairing the model’s ability to handle multi-object scenarios.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    SurgVLA-Bench is a benchmark framework for surgical VLA models. While timely, this work is limited to simulation, with associated issues of sim-to-real transfer, tissue deformation, etc. Further lack of a zero-shot baseline, inconsistency in reporting, and the lack of clarity around generalization prevent acceptance.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The rebuttal clarifies many of the reviewers’ concerns (data diversity, tissue realism, zero-shot baselines, etc.) While R1 raises valid concerns about sim-to-real transfer not being validated within this work, the rebuttal points out that Surrol has been shown to have good sim-to-real transfer in prior work. Moreover, with the potential for future work to separate surgical perception from surgical decision control, as in self-driving, this work provides a valuable benchmark for surgical decision control, which is arguably the more critical component for surgical autonomy.

    R1 also characterizes the work as a limited extension from Surrol. Unfortunately, the authors have no opportunity to respond to this point, which was not raised in the initial review, but I believe they would argue for the value of hierarchical task decomposition specifically suited for VLA evaluation, whereas Surrol focuses on atomic tasks.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    I recommend acceptance. The paper presents SurgVLA-Bench, a timely benchmark for evaluating vision-language-action models in surgical robotics. Although the work is primarily a benchmark rather than a new VLA architecture, it addresses a clear gap in the surgical robotics community by providing hierarchical tasks, standardized datasets, and evaluation of representative VLA models.

    The rebuttal addressed the main technical concerns raised by the reviewers. In particular, the authors clarified the scope of the simulator-based benchmark, the relevance and limitations of sim-to-real transfer, the current exclusion of deformable tissue tasks, the zero-shot baseline behavior, and inconsistencies in the reported SmolVLA results. Reviewers #2 and #4 support acceptance after rebuttal. Reviewer #1 remains concerned about simulation realism and clinical transfer, which is a valid limitation, but I view this as acceptable for an early benchmark contribution rather than a fatal weakness.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After reviewing the rebuttal letter and reviewers’ post-rebuttal comments, I recommend the paper for acceptance. Even though the technical novelty is limited, the proposed benchmark and evaluated baselines are suitable contributions.



back to top