List of Papers Browse by Subject Areas Author List
Abstract
Laparoscopic hepatectomy demands real-time identification of critical vascular structures, including Glisson’s pedicle, hepatic veins, and transient bleeding points, to enable pre-emptive vessel clamping and prevent life-threatening haemorrhage. These targets are frequently obscured and deformable, and in ultra-long procedures spanning thousands of frames they further suffer extreme scale imbalance and intermittent visibility, making memory-based propagation prone to temporal drift where a single corrupted entry cascades forward irreversibly.
Memory-based propagation dominates long-range video segmentation, yet existing methods treat all predictions equally when updating the memory.
Moreover, efficient intraoperative guidance demands minimal surgeon-model interaction, making autonomous quality control indispensable.
We present VigilSAM3, built on SAM3’s dual-system architecture, which introduces a hierarchical memory firewall that guards memory quality at pixel, class, and frame levels. A causal-Transformer scheduling agent selects among Trust, Correct, and Reset actions, with its policy refined via Group Relative Policy Optimization (GRPO) using class-aware rewards.
We further introduce Hepa-Vein, the first multi-class large-scale hepatectomy surgical video semantic segmentation dataset. Experiments on Hepa-Vein and DSAD demonstrate state-of-the-art performance, with pronounced gains on small, transient vascular structures.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1296_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{YaoXin_VigilSAM3_MICCAI2026,
author = { Yao, Xincheng AND Yang, Yijun AND Song, Zhuojin AND Guo, Kangwei AND Tao, Haisu AND Yang, Jian AND Zhu, Lei},
title = { { VigilSAM3: Reinforcement-Optimized Memory Gating for Ultra-Long Surgical Video Segmentation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents VigilSAM3, a reinforcement-learning-optimized framework for long-horizon surgical video segmentation, specifically targeting laparoscopic hepatectomy. The authors address the critical problem of “temporal drift” in ultra-long surgical videos by introducing a hierarchical Memory Firewall and a GRPO-based scheduling agent that dynamically chooses between Trust, Correct, and Reset actions. Furthermore, the paper introduces Hepa-Vein, a large-scale dataset with 21,734 annotated frames focused on critical vascular structures
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Technical Innovation: Using reinforcement learning (GRPO) to solve the sequential decision-making problem of “when to update memory” is a principled and effective approach to mitigating temporal drift. Clinical Relevance: The focus on small, transient structures like “Blood Points” (<0.1% of frame area) is highly relevant to surgical safety. Dataset Contribution: Hepa-Vein appears to be a high-quality, expertly annotated dataset that fills a gap in hepatectomy benchmarks. Strong Empirical Results: The model achieves state-of-the-art performance on both Hepa-Vein and DSAD datasets, particularly showing significant gains in challenging small-target classes.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Fairness of Baselines (Model Versioning): The proposed method is built upon SAM3, while many compared SOTA methods (e. g. , ReSurgSAM2, MedSAM2) are based on the older SAM2 architecture. It is unclear how much of the performance gain is attributed to the SAM3 backbone vs. the proposed RL-scheduling and Memory Firewall. Question: Can the authors provide results where the baseline models (like ReSurgSAM2) are implemented using the same SAM3 backbone for a fairer comparison?
Baseline Performance Discrepancy: In Table 1, the performance of MedSAM3 on “Blood Point” (39.32 Dice) is significantly lower than the proposed method (66.73 Dice). Question: Was MedSAM3 specifically fine-tuned for small-target detection on this dataset? Such a large gap suggests the baseline might not have been fully optimized.
Real-time Latency Peaks: While the average FPS (22.53) is reported , the system involves a heavy “System 2” detector for corrections. Question: What is the peak latency when the “Correct” or “Reset” actions are triggered? For real-time surgical guidance, the consistency of the frame rate is as important as the average speed.
There are minor typographical errors in some table labels (e. g. , “Glison” instead of “Glisson”)
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
no
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
VigilSAM3 is a technically sound paper with a clear clinical application and a strong new dataset. However, the lack of architectural parity in comparisons and the unaddressed latency peaks prevent a higher recommendation. I recommend a Weak Accept, pending the authors’ clarification of these points during the rebuttal.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
Long-range video segmentation usually depends on memory-based propagation methods like SAM2 and SAM3, but the memory can, over time, become corrupted through inaccurate predictions, causing drift in future frames. This paper proposes a way to tackle this drifting problem through memory gating, which observes the quality of predictions at the pixel, class, and entire frame levels before updating the memory. A scheduling agent based on a transformer continuously observes the quality of the memory and takes three different actions, and can even evict already stored memories from the tracker and reset if the quality worsens. The policy of the agent is refined through GRPO.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The motivation of the paper is clear. It shows a strong attempt to solve the limitations of current video segmentation methods when applied to ultra-long videos, especially in the surgical context.
- The paper is somewhat easy to follow, with good overall structure and organization.
- All components of the method seem to complement each other and appear necessary, with clear and justified explanations for their inclusion.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Major: Overall lack of related work on long-term surgical segmentation methods. Also, no mention of other works that address memory drift and memory management in video segmentation.
- Major: The evaluation is selective: the paper reports class-wise scores but omits an overall (full-image) metric. Furthermore, on the DSAD dataset, only 2 out of 8 classes are reported, raising concerns about fairness of the comparison.
- Comparison methods were also not optimised for rare classes. E.g. nnunet can be trained with class-weighted CE for improved performance on rare classes
- Needs clarification how video segmentation performance is being evaluated with datasets where annotation is not on complete video
-
The evaluated domain is very limited especially for general segmentation task. Need additional surgical domains (e.g., Cataract-1k, AutoLaparo, ,..)
- How is temporal continuity handled during the reset action? It would be helpful to also report temporal continuity/smoothness metrics.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Weak accept. Even though the motivation and methodology of the paper are strong, the selective evaluation, specified in weakness, raises concerns
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper proposed the VigilSAM3 Framework which introduce a hierarchical memory mechanism to control the quality of memory in SAM3.VigilSAM3 uses a Reinforcement Learning-based method to decide the actions of the memory control. This paper also propose a dataset for laparoscopic hepatectomy which consist of 39 videos.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Proposed the Hepa-Vein dataset with dense labeling; 2.Good quantitative results were achieved on both the Hepa-Vein and the external DSAD datasets. For the extremely small and intermittently occurring “Blood Point” category, a significant improvement in the Dice score was realized compared to existing methods. 3.By adopting the critic-free GRPO algorithm, the framework avoids the expensive computational overhead associated with approximating value functions in high dimensional long surgical video scenarios
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.This approach introduces excessive computational overhead by deploying expensive RL and detection modules to tackle visibility issues that classical methods solve more efficiently. For instance, kinematic priors provide a more elegant solution for occlusions, and the ByteTrack philosophy demonstrates that low-confidence detections still hold valuable temporal information. A more fundamental flaw lies in the handling of endoscopic camera ego-motion. Without Global Motion Compensation (GMC), treating corrected frames as persistent Anchored Memory creates a spatial mismatch as the lens moves, leading to catastrophic drift rather than stability. 2.The core of the “Hierarchical Memory Firewall” essentially reduces to standard residual fusion and MLP-based gating, offering limited novelty in spatiotemporal modeling. Given that recent medical imaging tasks have shown that Confidence-based Reset mechanisms can effectively mitigate memory drift in occluded scenarios, the complexity of the RL-based approach must be justified. A training-free baseline utilizing simple confidence-triggered resets is mandatory. This would prove that the GRPO agent’s performance stems from genuine causal reasoning over long sequences, rather than simply mimicking a static decision threshold. 3.A critical concern in the proposed architecture is its over-reliance on the perceived authority of System 2.By designating System 2 outputs as persistent anchor, the authors overlook the reality that detectors frequently produce false positives in noisy endoscopic environments. This design creates a positive feedback loop of errors; a single misdetection becomes a permanent reference point, ensuring that the memory remains permanently polluted. Without a verification or aging mechanism for these anchors, the system’s long-term stability is fundamentally compromised.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The clinical motivation of this work is commendable, and the Hepa-Vein dataset constitutes a substantial contribution to the MICCAI community. However, as a methodology-focused paper, the core logic appears increasingly fragile under rigorous scrutiny. I hope the authors can address these issues in the rebuttal:
1.Provide some training-free baseline using simple confidence-threshold resets to justify the indispensability of the RL agent. For examples, BoT-Sort, ByteTrack, and the other training-free methods like https://arxiv.org/abs/2602.19380.2.Explain or implement a Global Motion Compensation (GMC) mechanism to resolve the failure of static anchors caused by camera ego-motion. 3.Demonstrate that the Hierarchical Firewall and RL scheduling yield significant performance gains even when stripped of the parameter advantage of SAM3.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
In the rebuttal, the authors provided helpful textual clarifications, particularly regarding the low false-anchor rate, which partially alleviates my concerns about long-term error accumulation.
Author Feedback
Thanks for kind advice.
R1 (1) To isolate the SAM3-backbone effect, we built a SAM3-backed ReSurgSAM2-style baseline by preserving credible initialization and diversity-driven memory selection on the official SAM3 detector/tracker. It obtains Dice/IoU 68.14/55.93, while VigilSAM3 obtains 69.18/57.22, suggesting the gain is not merely from SAM3.(2) MedSAM3 releases image-level SAM3 LoRA training but no video memory-propagation recipe. We adapted it with the same first-frame prompt/tracking protocol as our SFT stage, without MF or Correct/Reset. At shorter clip length, MedSAM3 reaches Blood Pt. Dice/IOU of 57.79/46.29.The remaining gap is because pure tracking cannot recover once tiny Blood Points are missed or written inaccurately into memory. In VigilSAM3, Correct/Reset invokes System 2 to generate dense logits S_t conditioned on the existing prompt and draft D_t; S_t refines output and memory. No manual prompt or GT mask is used at inference. (3) On A800, forced Trust/Correct/Reset peaks at 36.3/154.9/143.5 ms. The latter two are sparse, so latency still controllable. (4) We will correct “Glison” to “Glisson”.
R2 (1) We will expand related work: STCN/AOT establish memory-based VOS; XMem designs sensory/working/long-term memory; SAM2Long/DAM4SAM address SAM2 error accumulation, distractors, and memory selection; SurgSAM2/ReSurgSAM2 study surgical SAM2 via frame pruning, credible initialization, and diversity-driven memory. We position VigilSAM3 as complementary: it learns Trust/Correct/Reset and gates writes at pixel/class/frame levels. (2) The two DSAD classes originally reported are vascular structures, matching Hepa-Vein’s goal; this was for vascular generalization rather than cherry-picking. We further evaluated 11 DSAD classes: mesenteric artery 74.65, intestinal veins 64.71, abdominal wall 84.21, colon 68.35, pancreas 41.53, spleen 68.99, stomach 82.58, small intestine 65.59, ureter 31.65, vesicular glands 45.74, liver 76.07, with overall Dice/IoU 63.14/52.68, which is competitive with SOTA methods. (3) VigilSAM3 uses no class-weighted segmentation loss; rare-class handling comes from class-aware GRPO reward and per-class memory gating. U-Net/nnU-Net with class-weighted loss obtain Blood Point Dice/IoU 26.05/19.74 and 28.32/21.97.(4) Hepa-Vein is extracted from long raw videos, but all retained benchmark frames are densely annotated; every evaluated frame has a GT mask, with no unlabeled/interpolated/pseudo-labeled frames. (5) For domain breadth, AutoLaparo gives Dice/IoU 87.26/80.91, close to LMS-Net (88/82), supporting generalization beyond hepatectomy and DSAD. (6) For temporal continuity, VigilSAM3 outperforms ReSurgSAM2 on Contour Jitter↓/t-IoU↑: 11.13 vs. 14.47, and 71.65 vs. 65.96; Reset does not introduce discontinuity.
R3 (1) The agent is lightweight; main overhead is occasional System-2 invocation, whose latency is bounded. We thank the reviewer for GMC. By adding lightweight GMC variant in the dense-prompt path: for an anchor at frame a, its dense logits are warped to frame t using H_{a->t} before memory conditioning, without changing SAM3 detector/tracker. This improves Dice/IoU from 69.18/57.22 to 69.77/57.69.We will consider this direction for extension. (2) Table 3 isolates SA, MF, and GRPO under the same SAM3 backbone. We added training-free Periodic Correct, Confidence Correct, and Confidence Reset baselines with the same SAM3 detector/tracker and prompt protocol. The best rule baseline is Confidence Reset, but is still 4.48/3.51 lower than VigilSAM3 in Dice/IoU, supporting that GRPO is not mimicking a static threshold and simple non-learning scheduling cannot replace the learned policy. (3) To test false anchors, promoted anchors have mean Dice 92.79 and false-anchor rate 1.63% (Dice<0.3). After anchor promotion, t-IoU is 74.39 and Contour Jitter is 7.08, showing no persistent error feedback loop. We will clarify that Reset removes recent contaminated memory and re-initializes the current state.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The reviewers generally found the paper clinically meaningful and technically promising, and all current recommendations are in the weak-accept range. At the same time, several important concerns were raised that should be addressed carefully in the rebuttal, as they may materially affect the final assessment.
First, the rebuttal should clarify the fairness of the experimental comparisons, especially the extent to which the reported gains come from the proposed memory-gating and scheduling design versus the use of a SAM3-based backbone while several competing methods are built on earlier SAM variants. In particular, the comparison to MedSAM3 and other strong baselines would benefit from clearer discussion of whether they were fully optimized under matched settings, especially for the rare and small Blood Point class.
Second, the necessity of the RL-based scheduling component should be more convincingly justified. One reviewer explicitly questioned whether simpler non-RL alternatives, such as confidence-threshold-based correction or reset rules, could already recover much of the benefit. A strong rebuttal should therefore clarify whether the gains of the GRPO agent go beyond what can be achieved by straightforward heuristic scheduling, and why the additional complexity is warranted.
Third, the evaluation protocol should be clarified and, if possible, strengthened. Concerns were raised regarding selective reporting, including the fact that only two classes are reported on DSAD and that no more global or overall metric is shown. The authors should explain the rationale for this reporting choice, clarify how evaluation is performed when annotations are not available for complete videos, and address whether the conclusions remain consistent under broader reporting.
Fourth, the real-time and deployment-oriented claims would benefit from a more detailed runtime analysis. Reporting only average FPS is likely insufficient when the method includes heavier Correct and Reset actions. It would help to clarify the latency associated with each action type, the frequency of these actions during inference, and whether occasional correction steps introduce problematic latency spikes.
In addition, the rebuttal should address the stability of the anchored-memory design. Reviewers raised concerns about whether treating “System 2” outputs as persistent anchors could create long-term error accumulation when the detector is wrong, and whether camera ego-motion may undermine the validity of persistent anchors in endoscopic videos. Even if these issues are already implicitly handled by the architecture, they should be explained more explicitly and convincingly.
Finally, the reviewers also noted some issues related to positioning and completeness, including limited discussion of prior work on memory drift and memory management in long surgical video segmentation, the need to clarify rare-class optimization in baselines, and several minor presentation issues.
Overall, the paper remains competitive, but the rebuttal should focus on demonstrating that the observed gains are genuinely attributable to the proposed method, that the RL component is necessary beyond simple heuristics, and that the evaluation and deployment claims are both fair and sufficiently substantiated.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The authors’ rebuttal includes excessive new experimental results, which is a direct violation of the rebuttal guidelines. As the guidelines state, such a violation should lead to rejection.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The responses addressed the main concerns about fairness issue of evaluation setting and baselines. Yet, new results on new dataset should not be added.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The reviewers generally consider the paper clinically meaningful and technically promising, and they also recognize the value of the dataset contribution. The rebuttal clarified several concerns regarding memory stability and parts of the evaluation protocol. Although there are some concerns are partially solve, considering the reviews and response the paper is slightly above the acceptance threshold.