Abstract
Expert assessment of robotic surgical skills provides feedback that trainees need but is expensive to scale, and wrong automated scores can mislead training. Whether general-purpose vision-language models (VLMs) can serve as reliable skill raters, and whether their errors stem from missing information or from failing to use the information they are given remains untested. We evaluate GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Flash as zero-shot raters on JIGSAWS, which pairs surgical video and da Vinci kinematics with expert modified OSATS Global Rating Scale (GRS) scores. We find that VLMs can rank surgical performances from video but cannot score them at expert level: the best performances are under-scored, the worst are over-scored, and predictions cluster near the middle of the 6 to 30 scale. Even the best setting, GPT-5.5 at 3 frames per second, reaches a Spearman correlation of 0.58, indicating only moderate agreement with the expert ordering. However, the errors do not come from missing motion information. Given a text summary of the kinematics instead of video, the VLMs agree with experts near zero (ICC ≤ 0.06), yet a linear regressor fed with the same summary reaches ICC = 0.44; and adding kinematic summaries to the video lowers GPT-5.5 to ρ = 0.39. Finally, showing the model expert’s score references spreads its predictions toward the expert score range, but does not improve the ranking. Overall, current VLMs can see useful surgical cues, but are not reliable standalone skill raters on dry-lab robotic tasks.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/SafeSurg_023.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=Pm2X8yfOxS
BibTex
@InProceedings{CheSiy_Can_MICCAISAT2026,
author = { Chen, Siying AND Cheng, Xianyi},
title = { { Can Vision-Language Models Safely Rate Robotic Surgical Skill? A Failure Mode Analysis with Video, Kinematics, and Expert References } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS pending},
month = {pending},
page = {pending}
}