Abstract
A frozen surgical foundation model such as LemonFM provides a strong backbone for endoscopic perception, yet the standard frozen backbone probing protocol discards both the spatial layout of the backbone’s feature map and the temporal structure already encoded in its learned embedding geometry. Self-supervised distillation trains these
embeddings so that cosine similarity reflects semantic similarity between frames, suggesting that a useful notion of temporal continuity already exists in the pretrained feature manifold and can be exploited directly rather than relearned from scarce downstream annotations. We introduce STAG,acompact Spatio-Temporal Affinity Graph that builds on this in sight with three lightweight modules operating on the frozen backbone’s learned geometry: it recovers spatial evidence through class-conditional slot pooling over the pre-pool token grid, replaces the classification head with a cosine prototype layer aligned with the backbone’s metric space, and performs non-parametric temporal label propagation over a video graph whose edges are the backbone’s own cosine similarities. Across four benchmark settings (Cholec80 phase recognition, Cholec80 and GraSP
tool detection, and CholecT50 verb recognition), STAG surpasses the LemonFM baseline on every task, improving accuracy by 9.5 pp on phase recognition and mAP by 14.4, 5.2, and 6.0 pp on Cholec80 tool, GraSP tool, and CholecT50 verb detection respectively, while adding only 1-2 M parameters and keeping the backbone fully frozen. An ablation attributes these gains to each component: the spatial and prototype components drive large gains across all tasks, while temporal propagation adds +6.5
pp on densely sampled Cholec80 phase recognition. On GraSP, whose keyframes are 35 s apart with minimal continuity, the temporal component contributes-0.2 pp (gains arise from spatial and prototype modules), confirming that temporal reasoning engages where continuity is present. Compared to MS-TCN, STAG outperforms learned parametric convolution on tool detection by wide margins.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/COLAS_019.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=B2JEbBA9AG
BibTex
@InProceedings{KheAli_STAG_MICCAISAT2026,
author = { Kheyrkhah, Alireza AND Saleknia, Amir Hossein AND Karimijafarbigloo, Sanaz AND Azad, Reza AND Merhof, Dorit AND Sulaiman, Alaa},
title = { { STAG: A Spatio-Temporal Affinity Framework for Surgical Video Understanding } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS pending},
month = {pending},
page = {pending}
}