Abstract
This work extends an instruction-tuned Vision–Language Model (VLM) framework for 2D surgical-tool keypoint estimation to joint hand–tool localisation using RGB and depth information. Accurate localisation of hand and surgical-tool keypoints is an important step towards computer-assisted understanding of surgical activities and future 3D pose estimation. Monocular RGB images provide limited geometric information, whereas depth supplies explicit spatial cues that are important for resolving overlap, relative position, and scale in 3D hand–tool configurations. We propose a parameter-efficient RGB–depth instruction-following framework that generates normalised 2D keypoint coordinates from an image and a task prompt. The pretrained VLM backbone is kept frozen and adapted using Low-Rank Adaptation (LoRA). Depth is incorporated using two patch-aligned strategies: a scalar depth bias and a lightweight learnable depth encoder. The learnable encoder models contextual relationships across depth patches and injects task-specific geometric features into the existing visual tokens without increasing the token-sequence length. Experiments on the synthetic POV-Surgery dataset show that the learnable depth encoder achieves the lowest hand and tool MPJPE compared with the RGB-only and depth-bias configurations. A paired statistical comparison across the evaluated images found a significant reduction in hand-keypoint error compared with RGB-only input ($p=0.0188$). These findings indicate that learned depth features improve fine-grained hand–tool localisation and provide an important foundation for extending the framework towards 3D hand–tool pose estimation.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/DEMI_026.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=BUeKsfbZa0
BibTex
@InProceedings{DuaKri_DepthEnhanced_MICCAISAT2026,
author = { Duangprom, Krit AND Bhattarai, Binod AND Lambrou, Tryphon},
title = { { Depth-Enhanced LoRA-Tuned VLMs for Hand and Tool Keypoint Detection in Surgical Scenes } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17276},
month = {pending},
page = {pending}
}