Abstract
Reliable reasoning about surgical instruments, which tool is active, where it sits, and how it is used, is fundamental to computer‑assisted laparoscopic surgery. Medical Visual Question Answering (Med‑VQA) promises a flexible natural‑language interface for such reasoning, yet surgical benchmarks like Cholec80‑VQA rely on rigid, templated questions, raising an under-explored concern: do these models genuinely understand the surgical scene and its instruments, or merely exploit narrow statistical patterns? We present a systematic robustness evaluation of Hulu‑Med, a generalist medical vision‑language model natively trained on surgical VQA, under three categories of textual perturbation centred on laparoscopic tool understanding: common sense probing of instrument appearance and geometry, answerability probing of tools absent from the scene, and semantic invariance probing under clinically equivalent rephrasings. Our three scales (4B, 7B, 14B), reveal that while baseline accuracy improves with scale (0.79 to 0.91), robustness does not: the smallest variant abstains most reliably on unanswerable questions (0.92), whereas the largest hallucinates confident answers on most of them yet is most consistent under rephrasing. These results expose a disconnect between in‑distribution performance and genuine instrument‑level understanding, raising concerns about Med‑VQA reliability for laparoscopic deployment under distribution shift.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/SAFER_018.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=cHqOcNUbSP
BibTex
@InProceedings{SaaAsi_Robustness_MICCAISAT2026,
author = { Saadat, Asir AND Linte, Cristian},
title = { { Robustness Evaluation of Surgical Medical Visual Question Answering Models Under Textual Perturbations } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17262},
month = {pending},
page = {pending}
}