Abstract
Off-the-shelf vision-language models (VLMs) are increasingly used as frozen models for surgical visual question answering (VQA), where task-specific fine-tuning is often impractical due to limited labelled data and compute. However, they frequently answer surgical questions with high confidence despite being incorrect, making them unsafe in high-stakes settings where a model should defer under uncertainty. To this end, we propose Neighbourhood Disagreement, a novel gradient-free abstention method for frozen VLMs in surgical VQA. Our method samples local experts by applying random Gaussian perturbations to the frozen model weights and uses disagreement among the resulting perturbed experts
as an epistemic uncertainty signal. The perturbation radius σ acts as a temperature-like control in weight space. Small radii preserve overconfident agreement, intermediate radii expose useful disagreement while retaining task competence, and overly large radii trigger an accuracy cliff as
expert behaviour collapses. The neighbourhood disagreement improves selective prediction by identifying cases where the frozen model should
defer. Across three surgical VQA datasets, CoPESD, Cholec80-VQA, and SSG-VQA, we find a narrow transferable operating regime for Qwen2.5-
VL-7B around σ ∗ ≈ 0.004, near the edge of competence. This regime consistently improves calibration by reducing Expected Calibration Error from 0.27/0.28/0.30 to 0.04/0.07/0.04 while also improving selective prediction, cutting the Area under risk coverage from 0.56/0.41/0.53 to 0.35/0.37/0.37 on CoPESD/Cholec80-VQA/SSG-VQA. Our results on 4 VLMs show that weight-space neighbourhood disagreement provides a strong uncertainty signal for frozen VLMs in surgical VQA. The source code and implementation are publicly available at https://github.com/pramitengox/knowing_when_to_defer.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/SafeSurg_012.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=BCCEBI5YFs
BibTex
@InProceedings{SahPra_Knowing_MICCAISAT2026,
author = { Saha, Pramit AND Thakur, Anshul AND Stotland, Mitchell A.},
title = { { Knowing When to Defer: Gradient-Free Abstention for Vision-Language Models in Surgical VQA via Neighbourhood Disagreement } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS pending},
month = {pending},
page = {pending}
}