List of Papers Browse by Subject Areas Author List
Abstract
Surgical video segmentation often operates on long image sequences for which dense annotation is prohibitively expensive, making sparse supervision more practical. Existing approaches suffer from temporal drift due to error accumulation, while formulations based on video object segmentation (VOS) or independent per-frame processing rely on restrictive assumptions or dense labels. Recent VOS foundation models offer strong visual representations, which are often per-object and tightly coupled with memory, limiting scalability and flexibility under domain shift in surgical videos. We propose a reliability-gated asymmetric dual-memory framework built on self-supervised visual representations of DINOv3. Our model accounts for both short-term temporal continuity and long-term semantic stability using a gated transient memory with bounded capacity and an evolving anchor memory that incrementally builds semantic representations without requiring complete first-frame class coverage. By decoupling memory from the encoder, our framework enables direct use of pre-trained vision foundation encoders and facilitates data-efficient adaptation under surgical video domain shift. Experiments on multiple surgical video datasets demonstrate improved temporal consistency and robustness compared to state-of-the-art baselines. Code is publicly available at: https://github.com/nsapkota417/D-GEM
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5645_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/nsapkota417/D-GEM
Link to the Dataset(s)
https://www.synapse.org/Synapse:syn27618412/wiki/616881
https://www.kaggle.com/datasets/newslab/cholecseg8k
https://opencas.dkfz.de/endovis/
BibTex
@InProceedings{SapNis_Sparsely_MICCAI2026,
author = { Sapkota, Nishchal AND Zhang, Yejia AND Zheng, Bofang AND Ma, Xianshi AND Shi, Haoyan AND Mariam, Yohannes AND Chen, Danny Z.},
title = { { Sparsely Supervised Surgical Video Segmentation with Reliable Asymmetric Dual Memory } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This manuscript presents D-GEM, a sparsely supervised surgical video segmentation framework with an asymmetric dual-memory design. It combines a Gated Transient Memory for short-term temporal consistency and an Evolving Anchor Memory for long-term semantic stability. A reliability-aware gating mechanism, based on prediction confidence and uncertainty, selectively updates memory to reduce error accumulation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Its main strength lies in demonstrating practical benefits, particularly reduced temporal drift and more reliable predictions in later frames. These advantages are supported by consistent improvements across multiple datasets.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.It would be beneficial if the introduction part could more explicitly relate the proposed components to existing lines of work, such as prior gated memory update strategies, semantic anchor or prototype-based representations, and approaches that model short- and long-term temporal dependencies. This would help readers better understand the motivation, task-specific design, and distinctions of the proposed method from previous work.
2.The evaluation is primarily based on quantitative metrics, while qualitative analysis (e. g. , visualization of segmentation results over time, comparison of temporal consistency, or representative success/failure cases) is limited. Such analysis would provide more intuitive evidence of the claimed improvements.
3.While module-level ablations are provided, several important factors, such as the number of anchors, memory capacity, gating thresholds, and fusion parameters, are not systematically analyzed. As a result, some design choices appear to be empirically selected, and their robustness could be further investigated. Moreover, the tables should indicate that it is recorded in %, such as Table 1, 2 and 3.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this work addresses an important problem and shows promising gains, the novelty and positioning against prior work are not yet sufficiently clear or convincing. In addition, the evaluation would be stronger with more qualitative analysis and a more systematic study of key design choices to better support the claimed advantages.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The response better clarifies the positioning of the proposed approach. The authors also commit to improving the introduction and adding more qualitative analysis, which addresses my concerns. Therefore, I am happy to raise my score.
Review #2
- Please describe the contribution of the paper
The paper proposes D-GEM, a sparsely supervised surgical video segmentation framework that decouples a pretrained DINOv3 encoder from a reliability-gated asymmetric dual-memory module. The Gated Transient Memory (GTM) maintains short-term context with bounded capacity and update gating, while the Evolving Anchor Memory (EAM) incrementally builds long-term class-specific anchors without requiring complete first-frame class coverage. Experiments on CholecSeg8k, EndoVis2018, and SAR-RARP50 show improved mIoU over several baselines, reduced temporal drift, and ablation evidence for the memory components.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) Clear and coherent architectural idea with explicit asymmetry between short-term and long-term memory. The division into GTM and EAM, along with different capacity and update rules, is well motivated for sparse supervision and long videos. Reliability gating and evolving anchors are concretely specified, including gating statistics (confidence and entropy), Top-K selection, merging by cosine similarity with threshold τ_sim, and eviction policies (Pages 4–5). These choices are reasonable for controlling drift and memory growth.
2) Strong empirical gains. In Table 1, with 10 labeled frames, D-GEM outperforms DINOv3Seg-V by +3.2 avg mIoU and SAM3[R] by +10.3 on SAR-RARP50. 3) Temporal drift analysis is thoughtful. Table 2 quantifies first-20% vs last-20% performance under varying supports; D-GEM maintains materially smaller degradation than SAM3[R], especially on SAR-RARP50.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Although I think the writing of this article and the methods proposed are generally satisfactory, there are still some issues that have not been clearly explained and need to be addressed.
1) the author mentioned in the introduction “surgical videos exhibit the frequent object appearance/disappearance, occlusions, visual corruption”, “Severe class imbalance further limits the effectiveness of first-frame–only (support frame) annotation schemes, as many key objects may be absent or underrepresented in the labeled frames”, but this is only a textual description, and as a reader, one cannot very intuitively appreciate the specific challenge. For example, what the author said, “as many key objects may be absent…” If there is a figure that can visually display this scene, such as which objects will be absent in the support frame, this can help readers understand better.
2) The description of the method is not very clear. In Section 2.1, the author mentioned that the pretrained DINOv3 ViT encoder would be used to generate patch-level features, but the mechanism explanation of how these features are stored in memory is not very clear. 2.1) For example, in GTM. I. The author mentioned “GTM maintains a compact set of patch-level tokens in recently processed consecutive frames under a fixed capacity. What does “K” refer to here? Is it the number of frames? II. It is also mentioned later that “confidence is measured as the mean maximum softmax probability while uncertainty is captured by the entropy of the predicted distribution”, on what dimension does the mean of confidence refer to here? Is it spatial mean? What is the predicted distribution and what is the entropy? I suggest that the author add some formulas to help readers understand better. III. “a frame-level prototype is formed by spatial pooling, and a Top-K subset of patch tokens is selected based on cosine similarity”, What is this spatial pooling? Is it average pooling or max pooling? Do the Top-K and capacity K here represent the same thing? IV. The gating strategy for whether each frame is accepted is very vague.
If I understand correctly, GTM retains the results of K_1 frames, and then K_2 patches are retained for each frame? I suggest the author rewrite this paragraph to facilitate readers’ better understanding.
2.2) The same problem also occurs in the description of EAM. I. “Given annotated frames, patch-level features are extracted and aggregated into class-conditioned prototypes, which serve as candidate anchors”, how are these patch-level features aggregated into class-conditioned prototypes? II. What is the capacity of EAM? If it is like the EAM limited to 2 anchors per class mentioned in Section 3, Then how are the 256 anchor tokens and 128 transient tokens used in the memory retrieval calculated? Shouldn’t the number of anchors be just 2 x number_class? How was this 256 obtained?
The entire Section 2.1 needs to be reorganized and accompanied by appropriate formulas and illustrations to help readers better understand.
3) Regarding the experimental part, if the understanding is correct, are SAM 2 and SAM 3 not fine-tuned? (Because there are no trainable parameters). I wouldn’t say the comparison is unfair, but the author needs to tone down his criticism of SAM2/SAM3 in the introduction. Namely, “state-of-the-art (SOTA) foundational VOS models that tightly couple temporal states with feature extraction (e.g., memory-conditioned encoder designs such as SAM2/SAM3 [2,13] video models) can reduce flexibility for domain adaptation, as pretrained representations are less easily reused or adapted independently under domain shift and limited supervision.” Since they were not trained on these datasets, SAM2/SAM3 still demonstrated very good zero-shot inference capabilities. At the same time, if it has not been fine-tuned, please explain why it is not fine-tuned? Such a comparison is more convincing.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this article does not describe the proposed method very clearly, its overall idea is good, and the experiments are relatively sufficient. If the author can address the questions I raised regarding the descriptive aspect of the method, I will maintain my current rating.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The author has tried to answer my questions and most of my questions have been addressed.
Review #3
- Please describe the contribution of the paper
This paper proposes D-GEM, a sparsely supervised surgical video semantic segmentation model, based on an asymmetric dual-memory design. The method decomposes temporal reasoning into a reliability-gated transient memory (GTM) and evolving anchor memory (EAM) for long-term semantic representations. The framework integrates the lightweight memory module with a pretrained DINOv3 encoder, enabling per-video adaptation under sparse supervision. Experimental results across multiple surgical datasets with different video lengths show consistent performance gain. And the temporal drift evaluation further demonstrates method robustness.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The separation between short-term (GTM) and long-term (EAM) memory is intuitive for surgical scenes. In particular, incremental semantic representations that do not require full class coverage within a single anchoring frame are important for dynamic object appearance. 2.The memory generation and fusion mostly rely on similarity-based operations, which are lighter weighted compared to encoding and cross attention of SAM2-style memory. 3.The results in Table 1 show that, even with only 1-label support, the proposed memory outperforms the SOTA models with a similar frozen encoder backbone, demonstrating the effectiveness of the memory mechanism. 4.Consistent performance gain on the experiments on three datasets with different lengths suggests good generalizability of the proposed asymmetric dual memory. 5.The evaluation of temporal drift by comparing the performance at the beginning and end of videos is a valid, clinically relevant experiment. This analysis provides deeper insights into robustness beyond mIoU.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The proposed model needs densely annotated support frames for best performance. 10 annotated frames per video at inference are a significant annotation cost, compared to no annotation or sparse prompts, as in SAM2/3.2.There are some ambiguities in the pipeline details. For example, the merging of new candidate prototypes with existing anchors in EAM is unclear. 3.Several important components are based on heuristic hyperparameters, such as thresholds of confidence and entropy in gating transient memory, memory capacity, anchor similarity threshold, and ramp-in scaling. The paper doesn’t include any sensitivity analysis. 4.The paper claims a benefit from “decoupling memory from the encoder”. However, as shown in Table 1, DINOv3Seg-V baseline with SAM2-style memory achieves similar performance gains after 10-frame fine-tuning, making it unclear whether the decoupling provides valid advantages over existing memory designs. 5.The evaluation mixes different adaptations, no parameter fine-tuning for SAM2/SAM3 and per-video fine-tuning for D-GEM and DINOv3-based models.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposed an asymmetric dual memory design tailored for surgical videos under sparse supervision. The experiments on three datasets show consistent performance gain compared with SOTA memory designs. And the temporal shift evaluation proves robustness.
The major concern is the requirement for dense annotations for per-video adaptation, which limits the model’s practical application. The experimental results are also partially confounded by per-video encoder fine-tuning, which appears to be a dominant factor in performance gains.
Overall, the paper is above the acceptance threshold due to its practical relevance and consistent improvements.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal solved major concerns. I keep my positive recommendation.
Author Feedback
We thank the reviewers for their thoughtful feedback and for recognizing the importance of sparse-supervised surgical video semantic segmentation (SSVSS) and the strengths of D-GEM, including its asymmetric dual-memory design for short- and long-term temporal reasoning, evolving anchors without complete first-frame class coverage, and architecture-decoupled memory formulation with robust performance and reduced temporal drift across videos.
Memory formulation and positioning [R1] We agree and will clarify in the Introduction to better position D-GEM. Our goal is not to claim an entirely new memory paradigm, but an SSVSS-specific design that separates short-term propagation from long-term semantic evolution. Unlike object-centric VOS methods that often rely on first-frame coverage or focus on either transient memory or static prototypes, D-GEM combines asymmetric reliability-gated and evolving-anchors based dual memory decoupled from the architecture. [R2] In EAM, we form class prototypes by pooling patch embeddings within labeled regions. Candidate prototypes are matched to the same-class anchors by cosine similarity, controlled by thresholds and budgets. “2 anchors/class” denotes anchor budgets, and “256 anchor tokens” refers to the dimension after Top-k retrieval, i.e., two anchors per class, each represented in 256 dim. [R2] For GTM, capacity K (different from top-k) denotes the number of recent (K) frames stored in memory, enforced by FIFO eviction. A frame prototype is computed by spatial average pooling over patch embeddings, and retained tokens are selected by cosine similarity to this prototype. Confidence is the spatial mean of the per-pixel/token maximum softmax probability, while uncertainty is the spatial mean entropy over the predicted class distribution. [R2, R4] We’ll revise Sect. 2 to improve the clarity.
Adaptation and comparison protocol [R2,R4] We agree that the discussion of SAM2/3 should be more balanced. Our intent was not to claim that SAM2/3 are weak, but to clarify that prompt-centric VOS models operate under different assumptions than SSSVSS. Accordingly, SAM2/3 were evaluated with periodic re-prompting, a practical sparse-prompt adaptation protocol for such models, while DINOv3-based variants use sparse fine-tuning enabled by their decoupled lightweight design. [R4] We emphasize that D-GEM already outperforms all baselines, including SAM2/3, in the strict 1-label setting (Table 1), so its key contribution does not depend on 10-frame adaptation. The 10-frame setting evaluates sparse adaptation in long videos, with only a small subset of frames labeled. D-GEM enables this by decoupling fine-tunable pretrained encoders/decoders from mostly training-free memory modules. [R4] We clarify that “SAM2-style memory” in DINOv3Seg-V refers to a single FIFO-style transient memory, not tightly coupled memory-conditioned encoding. Thus, DINOv3Seg, DINOv3Seg-V, and D-GEM are compared under identical encoder adaptation settings, isolating the effect of memory design. D-GEM outperforms DINOv3Seg-V under the same setups (frozen and fine-tuning), showing that its gains are indeed from our memory design.
Memory hyperparameters [R1,R4] These control the tradeoff among temporal persistence, compactness, and robustness to drift, and are linked to dataset properties. Sect. 3.2 quantifies these using easy-to-extract metrics for visual diversity (Pairwise Cosine Dist.) and frame stability (SSIM), which guide our memory/gating choices. Our revision will clarify these parameter-selection heuristics.
Qualitative visualization [R2] We included temporal drift evaluation comparing early vs. late video segment results, which the reviewers noted as relevant. Long surgical videos can contain many phase-dependent object appearances/disappearances. Thus, first-frame support can miss classes appearing later. We’ll add qualitative examples to show these. [R1] We’ll add qualitative results & correct the % notation in the table.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes an asymmetric dual memory design for sparse-supervised surgical videos, achieving good performance across three datasets and demonstrating robustness via temporal shift evaluation.While the work addresses an important problem with promising results and relatively sufficient experiments, its novelty and positioning against prior work lack clarity, and the presentation needs improvement. Major concerns of reviewers include the model’s reliance on dense annotations for per-video adaptation (limiting practicality) and performance gains partially confounded by dominant per-video encoder fine-tuning. Considering the cons and pros, the recommendation is Invite for rebuttal.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The authors provide a strong rebuttal regarding the concerns from reviewers, such as memory formulation and positioning. All reviewers are satisfied and recommend Accept. So the final decision is Accept.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes an asymmetric dual-memory framework for sparsely supervised semantic segmentation of surgical videos, featuring a gated transient memory and an evolving anchor memory, built on frozen DINOv3 representations. The reviewers raised concerns about positioning relative to prior gated-memory and prototype-based designs, the limited qualitative analysis and the absence of sensitivity reporting, and the reliance on dense per-video annotation in the best-performing setting. The rebuttal clarifies positioning, explains memory construction and gating semantics, and emphasizes that the proposed method outperforms strong baselines even in the strict one-label setting, with the sparser setting evaluating sparse adaptation rather than dense supervision. All three reviewers explicitly endorsed acceptance after the rebuttal, and the contribution is well aligned with the stated motivation. The camera-ready should incorporate the promised qualitative comparisons and additional ablations.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
All reviewers highlights the contributions of the submission and would like to accept.