Paper Info Reviews Author Feedback Meta-Review Back to top

List of Papers Browse by Subject Areas Author List

Abstract

Accurate surgical instrument segmentation in endoscopy is crucial for computer-assisted interventions, yet remains challenging due to frequent occlusions, rapid motion, and long-term instrument re-entry. While SAM3 provides a powerful spatio-temporal framework for video object segmentation, its performance in surgical scenes is limited by indiscriminate memory updates, fixed memory capacity, and weak identity recovery after occlusions. We propose ReMeDI-SAM3, a training-free extension of SAM3, that addresses these limitations through three components: (i) relevance-aware memory filtering with a dedicated occlusion-aware memory for storing pre-occlusion frames, (ii) a piecewise interpolation scheme that expands effective memory capacity, and (iii) a feature-based re-identification module with temporal voting for reliable post-occlusion identity disambiguation. Together, these components mitigate error accumulation and enable reliable recovery after occlusions. Evaluations on EndoVis17, EndoVis18 and CholecSeg8k under a zero-shot setting show mcIoU improvements of around 5.8\%, 8\%, and 2\% respectively, over vanilla SAM3, outperforming even prior training-based approaches. The code is available at: https://github.com/cgtuebingen/remedi-sam3.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5016_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/cgtuebingen/remedi-sam3

Link to the Dataset(s)

N/A

BibTex

@InProceedings{BunVal_ReMeDI_MICCAI2026,
        author = { Bundele, Valay AND Hosseinzadeh, Mehran AND Lensch, Hendrik P. A.},
        title = { { ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}

Reviews

Review #1

Please describe the contribution of the paper

The paper proposes ReMeDI-SAM3, a training-free extension of SAM3 for surgical video segmentation, focusing on improving temporal consistency and identity preservation under occlusions. The key contributions include: dual-memory design and memory expansion strategy and feature-based ReID.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method does not require retraining and can be directly applied to existing SAM3 models, which is practically valuable. The paper clearly identifies limitations of SAM3 under occlusion and proposes a structured memory design that directly targets these issues. The feature-based ReID with temporal aggregation is simple, interpretable, and aligns with standard practices in tracking and identity verification.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

1.The performance on certain classes in EndoVis2017 appears relatively weaker. It would be helpful to provide a more detailed analysis to better understand these gaps. 2.The proposed framework involves per-object memory, feature banks, and multi-scale similarity computation, which may introduce non-trivial computational overhead. However, the paper does not provide analysis on efficiency, scalability, or runtime cost. 3.The paper states that “just before occlusion, instruments often exhibit reduced visibility and thus lower reliability scores,” while also arguing that these frames carry critical identity cues for re-identification. This raises a concern: if the predictions in these frames are unreliable, it is unclear how the extracted appearance features can reliably represent object identity. 4.Occlusion recovery event relies on the objectness score from SAM3.If the underlying model produces incorrect occlusion signals, the higher-level dual-memory mechanism may be affected.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission does not provide sufficient information for reproducibility.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

N/A
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is a good architecture. The design is coherent and likely effective in practice. Additional analysis on robustness, efficiency, and design choices would further strengthen the work.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

The authors have adequately addressed most of the concerns raised and improved the explanation of several technical components. Therefore, I support acceptance.

Review #2

Please describe the contribution of the paper

This paper proposes a training-free, zero-shot extension of SAM3 for surgical instrument segmentation and tracking. The main contribution is a relevance-aware memory filtering mechanism combined with piecewise interpolation, which extends the effective memory capacity and enables feature-based re-identification of objects after disocclusion.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Scalable memory and occlusion handling: The proposed memory mechanism is well-designed to scale with longer sequences, and the inclusion of an occlusion-aware component enables re-identification of instruments after disocclusion, which is important for robust tracking.
- Applicability to complex surgical scenarios: The method is well-suited for settings with multiple instruments, where tracking and re-identification are challenging. Leveraging the SAM3 model provides a strong foundation for segmentation in such scenarios.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Potential limitations in complex scenes: While the re-identification mechanism is well-motivated, the method may still face challenges in scenarios with many similar instruments, where distinguishing between instances becomes difficult. Although operating room conditions are often relatively controlled, variations such as occlusions, instrument similarity, or subtle appearance changes may still affect feature consistency and lead to re-identification errors.
- Computational overhead and real-time feasibility: The approach builds upon SAM3, which is already computationally demanding, and further extends it with additional memory mechanisms and re-identification modules. This likely introduces additional overhead, raising concerns about real-time applicability in surgical settings. The paper does not provide sufficient analysis of computational cost (e.g., latency, memory usage), making it difficult to assess its practicality in time-critical environments.
- Increased memory and computational overhead: The occlusion-aware module relies on a buffer that stores past frames, which introduces additional memory requirements. This design may increase computational overhead and could limit scalability, particularly in long sequences or real-time surgical applications.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission does not provide sufficient information for reproducibility.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

Does the model need prior knowledge of the number of instruments present in the scene before processing begins? In addition, if instrument X disappears and a new instrument Y with a similar appearance later enters the scene, how does the occlusion-aware module distinguish between true re-identification and the introduction of a new object? In such cases, is there a risk that the two instruments may be treated as the same instance? (It is mentioned in the paper that “Both datasets provide annotations for seven instrument categories.”)
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents an interesting approach by leveraging a memory module and object re-identification to extend SAM3 for zero-shot segmentation and tracking. The idea is well-motivated, and the empirical comparisons demonstrate promising performance relative to existing methods. These aspects support a positive assessment and lead to a weak accept recommendation. However, several limitations remain (see weaknesses), particularly regarding robustness in complex scenarios and computational overhead, which should be addressed to strengthen the contribution.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

I thank the authors for the detailed rebuttal and for addressing most of my concerns. The additional clarification regarding computational overhead, scalability, and the class-level formulation of the task was helpful. Overall, I lean toward acceptance, although I still believe there are some remaining limitations:

1.The method may face difficulties in scenarios involving multiple visually similar instruments. Since the approach operates at the class level rather than full instance-level re-identification, distinguishing between different instances of the same instrument class remains an open challenge.

2.Although the additional overhead appears moderate relative to SAM3, the method still does not achieve real-time performance for practical surgical deployment and would benefit from further optimization in future work.

Review #3

Please describe the contribution of the paper

This paper proposes ReMeDI, a training-free extension to SAM3 tailored for surgical video instrument segmentation under long occlusions and re-entry. The method introduces a dual-partition memory (relevance-aware and occlusion-aware), a piecewise interpolation of temporal positional encodings to expand memory capacity, and a feature-based re-identification module with temporal voting. Across EndoVis17, EndoVis18, and CholecSeg8k, ReMeDI delivers consistent zero-shot gains over baselines.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1.Demonstrates that careful memory curation and lightweight design can push a general-purpose model (SAM3) to strong, application-specific performance, potentially reducing annotation and domain-adaptation needs. 2.Evaluations span three widely used surgical benchmarks (EndoVis17, EndoVis18, CholecSeg8k) with consistent improvements over SAM3 in a zero-shot setting.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

1.Comparison baselines for CholecSeg8k are missing. Only SAM3 is compared in Table 5.2.Missing implementation details. E.g. what are prompts for SAM-based baselines? Did you train the specialist models on your datasets? If so, what are the implementation details for these baselines? 3.The proposed model achieves SOTA performance in only a single category of EndoVis17 and two categories of EndoVis18.The authors should provide a more in-depth discussion to explain these specific results. 4.No statistical significance or variability reporting (e.g., per-sequence variance, confidence intervals) is provided.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

1.How are the initial prompts provided in all experiments? For “SAM3 (Mask, Zero-Shot)”, do you use ground-truth first-frame masks or automatically generated masks/boxes/points? 2.How do you handle multiple concurrent instances of the same instrument class? The ReID bank is per class; can it disambiguate same-class instances, or is the method fundamentally class-level rather than instance-level? 3.What policies bound the unconditional buffer and the feature bank sizes (e.g., max entries, recency decay, eviction)? Have you observed memory growth or contamination issues, and how are they mitigated? 4.Could you provide runtime/throughput and memory usage comparisons vs. vanilla SAM3 on the same GPU? What is the overhead from ReID and dual memories?
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

ReMeDI is an interesting, training-free enhancement of SAM3.However, considering incomplete implementation descriptions, a lack of reproducibility (no code), insufficient benchmarking on CholecSeg8k and other concerns mentioned above, I rate this paper as a 3.
Reviewer confidence

Very confident (4)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Reject
[Post rebuttal] Please justify your final decision from above.

After reviewing the rebuttal, I remain unconvinced by the experimental validation. The performance gains offered by the proposed method are marginal (e.g., a mere 0.06% improvement over SAM3) and limited to only 1-2 out of 7 categories. This marginal improvement does not demonstrate a significant contribution or the clear effectiveness of the method. Consequently, I lean toward recommending rejection.

Author Feedback

We thank the reviewers for constructive feedback. Below, we address main recurring concerns and provide additional clarifications. 1.Efficiency/scalability (R2-W2, R3-W2/W3, R4-Q3/Q4): SAM3 already maintains per-object memory by fusing per-object masks with image features; ReMeDI differs in using a larger memory bank and a different frame-selection strategy. On EndoVis18 with a single RTX 4090, ReMeDI incurs modest overhead compared to SAM3: 6,299 vs 6,010 MB peak GPU memory (+4.8%), 4,820 vs 4,767 MB average GPU memory (+1.1%), and 351.1 vs 329.7 ms/frame latency (+6.5%). The expanded memory bank (M=15 vs 7) adds only ~5% to memory-attention cost since compute is dominated by image encoder. To ensure scalability: unconditional buffer is a per-object FIFO with max_size=15 storing recent frames with reliability above τ_occ (only 7 recent frames needed for occlusion-aware memory) while ReID bank is capped at 15 entries/object. ReID is lightweight, triggered only during occlusion recovery, and reuses already-computed image features. 2.Per-sequence variability/per-class analysis on EndoVis17 (R2-W1, R4-W3/W4): Across 8 sequences, ReMeDI achieves 74.50 ± 14.13 mcIoU versus 69.04 ± 17.75 for SAM3, corresponding to a 20% reduction in standard deviation across surgeries. These sequence-level means differ slightly from the frame-pooled paper-table mcIoU values since sequences are weighted equally. The improvement is statistically significant (Wilcoxon p = 0.023, ReMeDI wins 7/8 sequences), with a 95% cluster-bootstrap CI of [+0.64, +8.13] for paired ΔmcIoU. Per-class analysis shows improvements on 5/7 classes (vs SAM3), while the remaining two are statistically tied. On EndoVis17, SAM3 performs slightly better on UP class whereas ReMeDI performs substantially better on challenging EndoVis18 UP sequences. Overall, ReMeDI ranks among the top two methods in 4/7 classes; remaining gaps mainly against supervised methods. Our ReID module prioritizes precision in difficult re-appearance cases, occasionally over-rejecting recoveries and lowering performance on some classes (GR). 3.Pre-occlusion reliability/objectness (R2-W3/W4): ReMeDI uses lower-visibility pre-occlusion frames to aid recovery since they often contain critical identity cues. However, they can have unreliable predictions, and to avoid propagating them, the ReID module rejects ambiguous recoveries. SAM3 objectness is used only as a recovery trigger; final identity assignment is determined by ReID, making recovery robust to false-positive triggers. Performance might be affected if SAM3 fails to re-detect after reappearance, which is relatively rare in practice. 4.Similar instruments (R3 optional, R4-Q1/Q2): ReMeDI performs class-level rather than instance-level segmentation. Both SAM3 and ReMeDI-SAM3 use ground-truth first-appearance masks as initialization prompts so the model already knows instrument classes present. If a newly appearing instrument belongs to an already-tracked class, it inherits the same class identity; otherwise, an extra prompt would be required. 5.Baselines, prompts, CholecSeg8k (R4-W1/W2): All zero-shot SAM-family rows use the same first-appearance GT-mask prompt. Specialist baselines in Table 1 are reported from their published supervised settings without modification. For CholecSeg8k, Table 5 compares ReMeDI against SAM3 under zero-shot setting on the full dataset; existing supervised methods train/evaluate on subsets and are therefore not directly comparable. We additionally evaluated PerSAM, TrackAnything as zero-shot baselines (full MA-SAM2 code unavailable) and found ReMeDI-SAM3 to perform better, consistent with EndoVis17/18 results. These comparisons can be included in the final version (as new results are not allowed in rebuttal). Overall, ReMeDI is a training-free refinement of SAM3 that provides statistically supported gains under surgical occlusion with moderate overhead. For reproducibility, we will release the code upon acceptance.

Meta-Review

Meta-review #1

Your recommendation

Invite for Rebuttal
Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The rebuttal should address three decision-affecting concerns. First, the missing computational and efficiency analysis—runtime, latency, and memory overhead relative to vanilla SAM3—raised by all three reviewers, since this is central to the claimed practical advantage. Second, the missing baselines on CholecSeg8k, the incomplete implementation details for SAM-based and specialist baselines including prompt setup and training, the limited per-category SOTA on EndoVis17/18, and the absence of statistical significance or variability reporting. Third, the robustness of the re-identification scheme: same-class instance disambiguation, distinction between re-entry and new objects, and the apparent contradiction between low pre-occlusion reliability scores and their use as identity cues.
After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

The paper proposes a training-free extension of SAM3 for surgical instrument segmentation, featuring relevance-aware memory filtering, an occlusion-aware memory, piecewise interpolation, and a feature-based re-identification module. The reviewers raised three decision-affecting concerns: missing computational and efficiency analysis relative to SAM3, missing baselines on one dataset, the absence of statistical significance reporting, and the robustness of the re-identification scheme. The rebuttal demonstrates modest overhead relative to SAM3, provides per-sequence statistics with a paired non-parametric significance test on the main benchmark, clarifies that the re-identification module rejects ambiguous recoveries to avoid propagating unreliable identity cues, and explicitly states that the module’s scope is class-level rather than instance-level. Two reviewers endorsed acceptance after the rebuttal; one maintained a Reject, arguing that the gains are marginal and limited to a subset of categories, which is a legitimate but secondary critique. The framework is clean, and the rebuttal addresses the most consequential concerns.

Meta-review #2

After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Reject
Please justify your recommendation.

After the rebuttal, two reviewers R2 and R3 accpet and one reviewer R4 reject. After reading all the reviews，I agree with R4 that the performance gain is limited and unconvincing. Besides, there are other concerns about the computational efficiency and insufficient experimental validation. So the recommendation is Reject.

Meta-review #3

After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

The paper presents a practically valuable training-free extension of SAM3 for surgical video segmentation, and reviewers recognized its dual-memory design, occlusion-aware recovery, and feature-based re-identification as useful for improving temporal consistency under long occlusions. Although there are concerns about computational overhead, robustness in visually similar instruments, and incomplete benchmarking details, the rebuttal provided additional efficiency analysis, variability/statistical evidence, and clarification of the class-level formulation and memory design. Overall, the method shows meaningful zero-shot gains with moderate overhead and sufficient practical value to support acceptance.

back to top

ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation

Author(s):