Abstract
Data scarcity is a persistent bottleneck for surgical AI, particularly when new
instruments are introduced and few annotated recordings are available. In this
work we investigate whether modern video-generation ‘‘world models’’ can act as
a synthetic data engine for surgical tool localisation and tracking. We study two
open-source backbones from NVIDIA’s Cosmos family: the general purpose Cosmos-Predict 2.5 and the domain-adapted Cosmos-H-Surgical, and evaluate each on a curated subset of CholecTrack20
under three regimes: zero-shot inference, LoRA fine-tuning and full supervised fine-tuning. Generation is performed
in Video2World mode, conditioned on the first two frames of each clip and a
structured text prompt describing surgery type, phase, tools and tool tracks.
We evaluate the resulting clips along two complementary axes: video quality
(FVD, LPIPS and three VBench dimensions) and domain quality for
downstream perception, measured by running a YOLO-v7 tool detector on both
real and generated videos and comparing bounding-box consistency and per-frame
track adherence at 10 fps. Across 367 test clips, Cosmos-H-Surgical achieves the better distributional realism
while Cosmos-Predict 2.5 offers the better paired perceptual
fidelity and the strongest tool-track reproduction. Overall, world models can plausibly render new surgical scenes,
but faithfully reproducing instrument geometry frame-by-frame remains
substantially harder than matching global video statistics.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/DEMI_029.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=GlxzZ3pG6x
BibTex
@InProceedings{XuMou_Exploring_MICCAISAT2026,
author = { Xu, Moucheng AND Caramalau, Razvan AND Choi, Justin AND Lee, Matthew AND Robu, Maria AND Luengo, Imanol AND Stoyanov, Danail},
title = { { Exploring World Models for Surgical Video Synthesis for Tool Localisation and Tracking } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17276},
month = {pending},
page = {pending}
}