Abstract

Query-based segmentation methods have shown promising potential for surgical instrument segmentation and recognition, which is essential for scene understanding and downstream tasks in computer assisted surgery. However, most existing approaches predominantly rely on per-frame predictions and overlook cross-frame temporal priors as well as temporal-consistency constraints. This limitation often leads to unsta ble query representations and suboptimal category recognition. In this paper, we propose TEDi, a Temporal memory-Enhanced and Denoising transformer for surgical instrument segmentation that addresses these is sues through Memory Search Enhancement and Temporal Consistency Denoising. The former introduces a query-level memory bank and a mem ory search enhancement encoder to retrieve discriminative representa tions from historical frames, enriching current-frame features. The latter constructs a temporally consistent reference as a cross-frame semantic anchor to suppress temporally unstable predictions and promote seman tic coherence across frames. Extensive experiments on two benchmark datasets, EndoVis 2017 and EndoVis 2018, demonstrate that TEDi con sistently outperforms state-of-the-art methods, highlighting its potential to further advance computer-assisted surgery. Our code is available at https://github.com/argon-xixi/TEDi.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1672_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: https://papers.miccai.org/miccai-2026/supp/1672_supp.zip

Link to the Code Repository

https://github.com/argon-xixi/TEDi

Link to the Dataset(s)

Endovis2017: https://endovissub2017-roboticinstrumentsegmentation.grand-challenge.org/; Endovis2018: https://endovissub2018-roboticscenesegmentation.grand-challenge.org/

BibTex

@InProceedings{YuaJia_TEDi_MICCAI2026,
        author = { Yuan, Jiahong AND Mi, Weiming AND Zhang, Tao AND Zhou, Haoyin},
        title = { { TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    They propose a memory bank and a memory-search encoder that retrieve target representations from historical frames, providing cross-frame semantic priors for robotic surgical tool segmentation. They also introduce a window-based temporal-consistency denoising decoder that constructs a temporally stable reference to enforce identity-aligned temporal correction, thereby suppressing unreliable query representations. They match/exceed baselines on two well-known and used datasets (EndoVis 2017 and 2018).

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The results appear to be decently better than existing methods and the current state-of-the-art. 2.The memory bank and encoder are a good contribution. Exploring the different queries for the memory bank was very interesting. 3.Good comprehensive evaluation on both the EndoVis datasets, identifying key recent papers.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The acronym TEDi for Temporal memory-Enhanced and Denoising transformer for surgical instrument segmentation is pushing it a little. I would suggest using a better acronym. 2.The video provided in the supplementary materials shows random switching for the tool classes multiple times, predicting it incorrectly. This should have been mentioned in the paper, as it is a severe limitation: the model should not change the tool class between frames when the tool is still clearly visible.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Please review the paper for potential typos, grammatical errors or missing punctuation. Also, add the short-hand title too.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I would have given an Accept (5), but the missing limitation apparent from the supplementary material is a severe weakness which should have been addressed, as it suggests the model class tracking is not robust.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper proposes TEDi, a temporal memory-enhanced and denoising transformer framework for query-based surgical instrument segmentation. Built upon the Mask2Former baseline, it introduces two core modules: (1) Memory Search Enhancement (MSE), which maintains a query-level memory bank from recent frames and uses a memory search enhancement encoder to retrieve cross-frame semantic priors for enriching current-frame query representations; and (2) Temporal Consistency Denoising (TCD), which constructs a temporally consistent reference via identity-aligned Hungarian matching and employs a cascaded denoising cross-attention decoder to suppress unstable or noisy query predictions across frames. The framework specifically targets the problem of category misclassification and temporal semantic flickering in surgical instrument segmentation. Experiments are conducted on EndoVis 2017 and EndoVis 2018 benchmarks, showing improvements over prior state-of-the-art methods.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    S1.Well-motivated problem formulation. The paper clearly identifies and articulates a practical and important problem in surgical instrument segmentation: while query-based models achieve good localization, they suffer from severe category prediction errors and temporal inconsistency. The empirical analysis (Fig. 3) convincingly demonstrates the two failure modes — high-confidence misclassified queries and low-confidence correct queries — which directly motivate the proposed modules.

    S2.Efficient query-level memory design. Unlike prior video segmentation methods that store dense pixel-level features, the proposed query-level memory bank is compact (storing N×d embeddings per frame) and avoids redundancy. The mask-aware gating mechanism (Eq. 1–2) that fuses spatial mask information into the query embedding space is a reasonable design that preserves spatial context in a compact representation.

    S3.Principled temporal consistency denoising. The TCD module provides a principled mechanism for enforcing cross-frame semantic coherence. The use of Hungarian matching to establish identity-aligned correspondences between the temporal reference and current-frame queries, followed by residual denoising cross-attention, is methodologically sound. The periodic reset every T frames to prevent error accumulation is a practical design choice.

    S4.Comprehensive experimental evaluation. The paper provides evaluation on two standard benchmarks (EV17 and EV18) with three metrics and per-category IoU breakdowns. The ablation study (Table 2) systematically evaluates the contribution of each component and the sensitivity to hyperparameters K and T. The query visualization analysis (Fig. 3) offers an insightful and convincing way to validate the effectiveness of the proposed modules at the query level.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    W1.Limited novelty of individual components. While the overall framework is reasonable, the individual components draw heavily from established techniques. Query-level memory banks have been explored in video object segmentation (XMem [7], RMem [29]). The Hungarian matching for temporal alignment is standard in tracking literature (e.g., TraSeTR [28]). The denoising cross-attention mechanism resembles DN-DETR (Li et al., “DN-DETR: Accelerate DETR Training by Introducing Query DeNoising,” CVPR 2022) and related denoising strategies for detection transformers. The paper does not sufficiently discuss or differentiate from these closely related works, particularly DN-DETR and its extensions, which should be cited and compared.

    W2.Missing detailed computational cost analysis. The paper reports parameter counts in Table 2 but does not report inference time, FLOPs, or memory consumption. Given that TEDi introduces a memory bank, Hungarian matching at each frame, and a multi-block denoising decoder, the computational overhead relative to the baseline is a critical concern for real-time surgical applications. The claim of being “efficient” (Section 4) is not substantiated.

    W3.Lack of temporal consistency metrics. The paper claims to improve “temporal stability” and “semantic coherence across frames,” yet only reports standard per-frame IoU metrics. No explicit temporal consistency metric (e.g., temporal consistency score, class flip rate, mismatch rate across adjacent frames) is reported. This is a significant gap between the claimed contribution and the evaluation protocol.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a relevant and practical problem in surgical instrument segmentation — temporal instability and category misclassification — and proposes a technically sound framework combining query-level memory retrieval with temporal denoising. The experimental results show consistent improvements over the baseline and prior methods, and the query-level visualization analysis (Fig. 3) is particularly insightful.

    However, several factors prevent a stronger recommendation: (1) The individual technical components (memory bank, Hungarian matching, denoising attention) are largely borrowed from existing works (XMem, DN-DETR, etc.) without sufficient novelty discussion; (2) Important experimental details are missing, including computational cost analysis and failure case discussion.

    The rebuttal could strengthen the paper by: (a) adding temporal consistency metrics, (b) providing runtime/FLOPs comparison, (c) discussing relationship with DN-DETR explicitly.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper proposes TEDi, a temporal query-based surgical instrument segmentation framework built on Mask2Former, with two added components: a query-level memory bank for retrieving historical query representations and a temporal consistency denoising module for stabilizing cross-frame semantics. The method is evaluated on EndoVis 2017 and 2018 and reports improved segmentation performance over several recent baselines.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses a relevant problem in surgical vision, namely improving temporal stability for query-based surgical instrument segmentation rather than treating frames independently. 2.The method is clearly structured, with two intuitive modules—memory search enhancement and temporal consistency denoising—that directly target category confusion and temporal inconsistency. 3.The experiments are reasonably comprehensive, including comparisons on two EndoVis benchmarks, ablations, and qualitative analyses of query behavior.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The paper does not sufficiently clarify what is fundamentally new beyond prior temporal surgical segmentation methods. The claimed gains seem to come from better temporal use of historical queries and denoising, but closely related motivations—using temporal context, improving consistency, and leveraging cross-frame information—have already been explored in the cited surgical literature. The paper would be stronger if it more explicitly distinguished its contribution from MATIS and LACOSTE at the formulation level, rather than mainly through performance improvements. 2.The paper also does not adequately discuss its trade-off relative to recent SAM2-style methods. SAM2-related approaches typically rely on prompts, which makes them less directly comparable as fully automatic category-level segmenters, but they offer stronger generalization beyond a fixed closed set of instrument classes. In contrast, TEDi is designed for predefined instrument categories on EndoVis, so its improvements are demonstrated only in a label-fixed setting, and its ability to generalize to unseen instruments or broader surgical scenes remains unclear. A stronger paper should explicitly discuss this limitation and clarify the scope in which TEDi should be preferred over promptable segmentation frameworks such as SAM2 or Surgical SAM2.3.The paper lacks an efficiency analysis, making the reported gains harder to interpret. Although TEDi reports better accuracy, the paper does not provide a clear comparison of parameter count, computational cost, inference speed, or memory overhead relative to the baseline and prior methods. Since TEDi introduces extra memory retrieval and temporal denoising components, it remains unclear whether the improvement comes from a better design or simply from a larger and more computationally expensive model. This is an important omission for surgical video applications, where efficiency is often highly relevant. 4.There is also a presentation issue in the qualitative figures. Some visual results appear stretched or distorted, suggesting that the image aspect ratio may have been altered during figure resizing. This hurts readability and makes qualitative comparison less reliable, especially in a segmentation paper where boundary shape and instrument geometry matter.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Point 1 and 3 in weakness are major concerns.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors have solved my major concerns. I would raise to Accept.



Author Feedback

1.Novelty: We appreciate the reviewers’ request to clarify our core contributions. Our method, TEDi, introduces a systematic “complement-and-denoise” paradigm that addresses limitations in existing work through three key distinctions. First, regarding temporal instrument segmentation methods, current methods like MATIS and LACOSTE fail to simultaneously manage information deficiency and feature-level semantic noise. MATIS operates at the backbone level without query enhancement, while LACOSTE propagates queries without feature-level semantic noise suppression, leading to accumulated errors. TEDi bridges these gaps: the MSE module retrieves historical context to compensate for occlusions, while the TCD module suppresses the categorical noise inherent in temporal propagation. Second, regarding Video Object Segmentation (VOS) methods like XMem or RMem, our approach is significantly more efficient. Unlike VOS frameworks that store dense, pixel-level features, our memory bank utilizes compact, instance-aware query representations specifically tailored for the requirements of surgical instrument tasks. Third, regarding DN-DETR, our TCD module is fundamentally different in function and application. While DN-DETR is a training-only strategy using random noise for convergence, TCD is a permanent architectural component. It uses prior denoised queries as stable cross-frame semantic anchors during both training and inference without introducing random noise. The synergy of this design is empirically validated in Figure 3: MSE increases the retrieval of correctly classified queries (Fig. 3b), while TCD prunes the misclassified ones (Fig. 3c). We will clarify these distinctions in the revised manuscript.

2.Efficiency analysis: An efficiency analysis have been conducted on one NVIDIA A100 GPU with 512×512 input resolution. TEDi has 60.58M parameters and 78.92 GFLOPs, compared to the Mask2Former baseline’s 56.91 GFLOPs. Notably, TEDi reuses query representations from previously encoded frames during temporal inference, so only the newly arriving frame requires backbone encoding. This yields an inference latency of 57.96 ms/clip, where the memory bank contributes only approximately 5.5 ms of additional overhead, with a peak GPU memory footprint of 666.1 MB. Among prior methods, TEDi has lower complexity than ISINet(264 GFLOPs) and comparable to QPD(71.38 GFLOP) and S3Net(88.04 GFLOP).

3.Temporal consistency metric: To further quantify the temporal stability illustrated in Fig. 3, we introduce the Class Flip Rate (CFR), defined as the proportion of matched instrument instances whose predicted category changes between consecutive frames. On EV18, TEDi reduces CFR from 11.4% (baseline) to 4.8%, validating that our MSE and TCD effectively suppress inter-frame category flickering through cross-frame memory retrieval and temporal-consistency denoising.

4.Limitations: Thank you for point these out and we agree that TEDi improves category prediction consistency over prior methods but does not completely eliminate temporal category flickering, as observed in the supplementary video. The main failure cases occur when prolonged occlusion, or when instruments remain visible but undergo significant appearance variation, causing our MSE to retrieve unreliable representations, and for rare instrument categories with limited training samples, which remain more susceptible to misclassification.

5.Compare with sam2: SAM2 acknowledges difficulty with nearby similar-looking objects(Ravi et al., 2024) and TEDi outperforms SurgicalSAM2(Liu et al., 2025) on the main EndoVis metrics under fully automatic evaluation. TEDi is preferable when automatic, closed-set category-level prediction is required, while SAM2 offers better open-set generalizability. We will clearly mention this in the conclusion of the revised paper.

6.Naming and figures: We appreciate these suggestions and will adjust all qualitative figures to correct aspect ratios in the revision.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The rebuttal should clarify the technical novelty over prior temporal-memory and denoising approaches to surgical instrument segmentation, and provide the missing efficiency analysis (inference time, FLOPs, and memory overhead) since parameter counts alone are insufficient to substantiate the practical-deployment framing. The rebuttal should also report temporal-consistency metrics that align with the paper’s workflow motivation, and include a substantive discussion of failure modes and limitations.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper proposes a temporal memory-enhanced and denoising transformer for surgical instrument segmentation on the standard EndoVis benchmarks. The original concerns centered on technical novelty relative to memory-based VOS methods and query-denoising approaches, the absence of an efficiency analysis, and the lack of a temporal-consistency metric aligned with the workflow motivation. The rebuttal articulates clear distinctions: a compact query-level memory rather than dense pixel-level memory, and a permanent temporal-consistency denoising component that uses prior denoised queries as anchors, rather than a training-only random-noise denoising scheme. The reported efficiency profile substantiates the efficiency framing, and the newly reported class-flip-rate metric is consistent with the workflow motivation. Two reviewers explicitly endorsed acceptance after the rebuttal; the third did not update but raised mainly minor concerns. The contribution is coherent and well-validated.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After reviewing the rebuttal letter and reviewers’ post-rebuttal comments, I recommend the paper for acceptance, as all the major concerns were addressed by the authors in the rebuttal period.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The rebuttal effectively addressed the reviewers’ concerns, providing sufficient clarification and additional evidence where requested. Following the rebuttal, all reviewers expressed support for acceptance, indicating a clear consensus that the paper meets the standard for acceptance at MICCAI.



back to top