List of Papers Browse by Subject Areas Author List
Abstract
Vessel safety in endoscopic surgery, particularly in Endoscopic Submucosal Dissection (ESD), is currently supported by segmentation-based visual guidance, which provides spatial awareness but cannot anticipate dangerous vessel–instrument interactions before they occur. To address this limitation, we formulate vessel–instrument interaction anticipation as a new task that predicts the upcoming interaction state from a short observation window of frames and predicted segmentation masks. This task presents a fundamental tension between observation reliability and temporal sufficiency, as both visual and segmentation-derived inputs are inherently noisy, yet the brief observation window offers no margin to absorb such noise, with different interaction categories further exhibiting heterogeneous temporal evidence patterns.
To address these coupled challenges, we propose a hierarchical framework that explicitly decouples reliability estimation from temporal reasoning. A cross-modal latent recovery module enforces consistency between visual and geometric observations through an information bottleneck, learning a shared representation that prevents segmentation noise from propagating into temporal reasoning. A dual asynchronous memory module separates transient dynamics from stable interaction prototypes through differentiated admission and retrieval strategies, enabling robust temporal reasoning under unreliable observations.
We construct the first multi-case annotated ESD dataset for this task and demonstrate that the proposed framework achieves 70.80% Macro F1 at 97.82 FPS, confirming both strong anticipation performance and real-time clinical applicability.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2613_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/med-air/EndoInteractionAnticipation
Link to the Dataset(s)
N/A
BibTex
@InProceedings{CheYue_From_MICCAI2026,
author = { Chen, Yueyao AND Wang, Kai-Ni AND Yip, Hon-Chi AND Dou, Qi},
title = { { From Perception to Anticipation: Forecasting Vessel–Instrument Interactions in Endoscopic Surgery under Unreliable Observations } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper introduces a novel task of vessel–instrument interaction anticipation in endoscopic submucosal dissection (ESD), aiming to predict future interaction states (no interaction, approaching, touching) from short observation windows. This shifts the focus from retrospective recognition to proactive anticipation, which is clinically meaningful. To support this task, the authors present a new annotated multi-case dataset consisting of 30 real ESD procedures with clip-level annotations. Methodologically, the paper proposes a framework that explicitly addresses noisy and unreliable observations, a common issue in endoscopic vision.
The approach combines three key ideas: (1) a cross-modal latent recovery (CLR) mechanism to enforce consistency between visual and geometric features, (2) an explicit separation of reliability estimation and temporal modeling, and (3) a dual asynchronous memory module that captures both short-term dynamics and long-term trends, guided by a reliability score.
Extensive experiments suggest improved robustness to segmentation noise and demonstrate the feasibility of real-time inference.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper addresses a clinically relevant and well-motivated problem. Anticipating vessel–instrument interactions has clear implications for preventing complications such as vessel injury during ESD, making the shift from perception to anticipation highly meaningful.
A major strength is the introduction of a new task and dataset, which provides a foundation for future research. The task is clearly defined and distinct from existing work on phase detection or segmentation.
The work also stands out by explicitly modeling observation uncertainty, rather than relying on end-to-end learning to implicitly handle noise. The separation between reliability estimation and temporal modeling is conceptually clean and well motivated. Methodologically, the combination of cross-modal consistency and dual memory mechanisms is interesting and reasonably well aligned with the problem structure. The empirical results are convincing, with consistent improvements over baselines, particularly under unstable or noisy conditions.
Finally, the demonstrated real-time performance strengthens the practical relevance of the approach.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
A key limitation is the insufficient description of the annotation process, which is particularly critical given that a new task is introduced. Important details such as annotator expertise, annotation protocol, handling of temporal transitions, and inter-annotator agreement are missing. This raises concerns about the reliability and reproducibility of the dataset.
Related to this, the paper lacks transparency and reproducibility in several aspects. Core components such as the CLR module, reliability score computation, and gating mechanisms are described at a high level but not specified in sufficient algorithmic detail. Additionally, the main methodological figure 2 is very hard to understand and visually overloaded. The temporal and/or logical path remains intransparent. Having this complexity in mind, it is a major weakness that there is no statement regarding code and/or data availability.
Another concern is the reliance on automatically generated vessel segmentations (SAM2) as a core input to the proposed framework. While the paper explicitly aims to be robust to segmentation noise, no quantitative evaluation of segmentation quality is provided. Such a validation would provide an important reference for contextualizing the reliability of the input signal. Without this information, it is difficult to disentangle whether the reported robustness primarily arises from the proposed cross‑modal and memory mechanisms, or whether it benefits from already sufficiently accurate upstream segmentation. A quantitative assessment or at least a discussion of expected segmentation quality would strengthen the methodological clarity and interpretability of the results.
Finally, the evaluation is restricted to a single, internally collected dataset, with no external validation. This makes it difficult to assess robustness across different clinical settings, devices, or institutions.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a strong conceptual contribution with clear clinical motivation and introduces a novel and relevant task. The idea of explicitly modeling reliability and combining it with temporal reasoning is promising, and the empirical results are encouraging.
However, several critical issues limit the current contribution. In particular, the lack of a detailed annotation protocol, the absence of reproducibility information, and the lack of validation of the underlying segmentation quality weaken the overall reliability of the work.
With improved transparency, stronger validation (especially of the segmentation backbone), and clearer methodological specification, the paper could become a solid contribution.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal substantially addresses the main concerns raised in my initial review. Specifically: (1) The annotation process is now clearly specified, including annotator expertise, a concrete labeling rule based on the 4th-s target frame, dataset splits at case level, and quantitative inter-annotator agreement (94.4% / Cohen’s κ=0.90). This resolves my central concern regarding dataset reliability and reproducibility. (2) The methodological clarifications provide the missing algorithmic detail: the formulation of the alignment reliability term, the distinct roles of the three gating signals (ψ_t, r_t, λ_t), the composition of the 10-D geometric descriptor, the layer-selection ablation for f_t^(8), and the sensitivity of key hyperparameters. Together with the explicit commitment to release code, weights, and the annotation protocol, the reproducibility concern is largely mitigated. (3) The segmentation quality is now quantified (Dice/IoU = 84.64/73.37), and the differential F1 drop under visually unstable clips (8.17 for the proposed model vs. 32.84 for the baseline) provides convincing evidence that the reported robustness stems from the proposed reliability-aware mechanisms rather than from clean upstream input. This directly addresses my methodological concern about disentangling segmentation quality from model contribution. A remaining limitation is the absence of external validation across institutions or devices, which the rebuttal does not address. Given the scope of a conference paper and the strength of the conceptual contribution, the clear clinical motivation, the introduction of a novel and well-defined task, and the now-clarified methodology, I consider this an acceptable limitation to be discussed in the final version. Overall, the paper makes a meaningful contribution by shifting from retrospective recognition to proactive anticipation of vessel–instrument interactions, and the rebuttal has adequately resolved the main issues raised. I therefore recommend acceptance, conditional on the authors incorporating the clarifications and the committed code/data release into the camera-ready version.
Review #2
- Please describe the contribution of the paper
The paper introduces the task of vessel–instrument interaction anticipation in ESD procedures, aiming to predict future interaction states from short video clips with noisy segmentation masks. To achieve this aim, it proposes a framework combining: A cross-modal latent recovery (CLR) module to enforce consistency between visual and geometric features; A dual asynchronous memory module that separates short-term and long-term temporal features, with reliability-aware updates.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Clinically relevant, Anticipating instrument–vessel interactions before contact is a meaningful step toward proactive surgical safety systems. 2.The emphasis on robustness to noisy inputs is well-motivated and practically relevant. 3.The combination of cross-modal learning and temporal modeling is logically structured.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper is difficult to follow, with many concepts, variables, and modules introduced without sufficient explanation: 1.The core prediction task is not clearly specified early on. It is not immediately clear whether this is a standard classification problem, a sequence prediction task, or a forecasting problem. 2.Several terms, equations and components are introduced with insufficient intuition or justification, making it hard to understand how they contribute to the final prediction. For example, (a) the intermediate feature f(8) was selected for feature fusion, was other selections and impacts well studied? (b)The cosine similarity computation is inconsistent between the image and the text.(c) It is not clear how a_vis and a_align are derived. (d) Not all the components of the 10D geometry descriptor listed. (e), what is anticipation signal, and how it was computed (the classification results or computed on top of it?) (f) Hyperparameters such as memory sizes, thresholds, and fusion weights are not well analyzed. It is unclear how sensitive the method is to these choices. 2.The dataset is not well explained. what is the statistics of each class, how the dataset is split (randomly or clips from the same videos in the same set etc). 3.How noisy was the segmentation result, and how was cases with long periods lacking a reliable reference handled? 4.How was the instrument tip extracted? Overall, the lack of clarity significantly hinders understanding and weakens the paper’s impact.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper tackles a relevant and potentially impactful problem: anticipating vessel–instrument interactions for surgical safety, and proposes a reasonably structured framework combining cross-modal learning and temporal modeling. However, I recommend weak rejection due to poor clarity and insufficient explanation. The paper does not clearly define the learning problem formulation early on. Important concepts are not rigorously explained. Many components are introduced in a dense and abstract manner, with limited intuition or concrete examples. As a result, it is difficult to fully understand or reproduce the method.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
While the rebuttal provides additional implementation details and clarifies several definitions, my primary concerns regarding clarity, rigor, and reproducibility remain insufficiently addressed.
First, many essential methodological components are still described only at a high level without enough formalization or justification. The clarification of the 10-D geometry descriptor is also still incomplete, as the exact descriptor formulation and normalization procedures are not fully specified.
Second, the rebuttal confirms that several critical experimental details were absent from the original submission rather than demonstrating that the paper itself is sufficiently self-contained.
Third, the robustness claims remain only partially convincing. While the authors report segmentation Dice/IoU scores and relative degradation comparisons, there is still limited analysis of failure cases involving prolonged unreliable segmentation, severe occlusion, or unstable instrument visibility. In particular, manually annotating the instrument tip weakens the practical realism of the setup and leaves uncertainty regarding how the full pipeline would perform under fully automated conditions.
Overall, although the rebuttal improves understanding of the method, I do not believe it sufficiently resolves the underlying concerns about clarity, completeness, and experimental rigor. Therefore, my recommendation remains rejection.
Review #3
- Please describe the contribution of the paper
Formulation of a Novel Anticipation Task: The paper effectively shifts the paradigm from reactive perception to proactive safety by formalizing the Vessel-Instrument Interaction Anticipation task. Unlike conventional segmentation-based methods that provide only instantaneous spatial awareness, this new task focuses on forecasting potential risks within a short temporal window. This transition is highly relevant for clinical settings where early warning systems are critical for preventing intraoperative complications.
Robust Hierarchical Framework: To address the challenge of noisy surgical data, the authors propose a well-structured hierarchical architecture. The Cross-modal Latent Recovery (CMLR) module is particularly noteworthy for its use of an information bottleneck to filter out segmentation noise, ensuring that only reliable features are used for reasoning. Furthermore, the Dual Asynchronous Memory (DAM) module cleverly decouples transient instrument dynamics from stable interaction prototypes, enabling robust and accurate temporal forecasting even under unreliable observation conditions.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The most significant strength of this paper is the formalization of the Vessel-Instrument Interaction Anticipation task. While existing research in surgical AI has largely focused on real-time segmentation and perception, this work proactively addresses surgical safety by forecasting potential risks before they occur. Shifting the focus from reactive detection to predictive anticipation is a highly meaningful contribution that directly addresses the need for early warning systems in complex procedures like Endoscopic Submucosal Dissection (ESD).
The proposed hierarchical architecture is also technically impressive, particularly in its ability to handle noisy and unreliable surgical data. The Cross-modal Latent Recovery (CMLR) module effectively utilizes an information bottleneck to filter out noise from segmentation masks, ensuring that the temporal reasoning process is based on reliable features. Furthermore, the Dual Asynchronous Memory (DAM) module is a clever solution for modeling the heterogeneous nature of surgical interactions. By decoupling transient instrument dynamics from stable interaction prototypes, the framework maintains high predictive accuracy even when frame-level observations are limited or degraded.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
While the task of anticipation is novel, the reported Macro F1 score of 70.80% suggests that the system may still produce a significant number of false alarms or missed detections. In a high-stakes surgical environment like Endoscopic Submucosal Dissection (ESD), a 30% error margin could lead to alarm fatigue for surgeons or, more critically, a failure to prevent actual vessel damage. The authors should have provided a more detailed failure case analysis to discuss how these errors would be managed in a real clinical setting.
The dataset composition also raises concerns regarding the generalizability of the proposed framework. The study utilizes a newly constructed multi-case ESD dataset, but the paper does not specify the diversity of the surgical equipment or the variations in anatomical conditions across different patients. Since the appearance of vessels and the handling of instruments can vary significantly between surgeons and hospital settings, the lack of cross-center validation or a more diverse testing set limits the proof of the model’s robustness.
Another weakness lies in the definition of the anticipation window. The paper mentions predicting upcoming states from a short observation window, yet it does not clearly define the exact lead time or the prediction horizon. Without a rigorous analysis of the trade-off between the prediction horizon and accuracy, it is difficult to evaluate whether the system provides enough reaction time for a surgeon to actually alter their movement and avoid an accident.
Finally, although the dual asynchronous memory module is designed to handle temporal reasoning, its superiority over simpler temporal models like Transformers or LSTMs with attention mechanisms is not sufficiently justified through extensive ablation studies. The authors should have compared their memory module against a wider range of state-of-the-art temporal architectures to clearly demonstrate that the added complexity of dual asynchronous processing is necessary for achieving the reported performance.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposes a practical and original framework for real-time risk anticipation that goes beyond simple perception by utilizing an information bottleneck and dual asynchronous memory to overcome noise and temporal uncertainty in surgical videos.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers for their constructive feedback. R1 appreciated the “conceptually clean” decoupling and “convincing” results; R2 valued a “meaningful step” toward proactive surgical safety; R3 recognized the “highly meaningful contribution” and “technically impressive” architecture. Responses below:
1.Dataset construction & annotation (Meta, R1, R2, R3): (i) Dataset. The dataset contains 3330 clips from 30 cases across four sites (esophagus/stomach/colon/rectum: 310/400/2200/420 clips) and three classes (no_interaction/approaching/touching: 1650/960/720 clips). All cases were split at the case level into 22 training and 8 testing cases (2430/900 clips), avoiding within-video leakage while preserving label/site distributions. (ii) Annotation. Two trained annotators with 4+ years of endoscopic research experience independently labeled all clips, using the visible vessel–instrument relation at the target frame. For state-changing clips, the label follows the state at the 4th-s target frame. Initial agreement was 94.4% with Cohen’s kappa=0.90, and disagreements were resolved by consensus.
2.Task formulation & lead-time (R2, R3): The task is a forecasting problem formulated as 3-class classification of the future interaction state. The model takes 16 frames from a 3-s observation window and predicts the state at the 4th-s target frame, yielding Δt=1s lead time. The anticipation signal in Fig.4 is the class-wise softmax probability on f^m (Eq.1), and its touching curve rises 0.5–1s before contact, supporting timely preventive response.
3.Method clarifications (Meta, R1, R2): (i) CLR.
- f_t^(8) is an intermediate ViT-B layer preserving spatial detail before semantic abstraction. Layer ablation over {6, 8, 10, 12} confirms layer 8 is optimal, outperforming the second-best setting, f_t^(10), by 1.73 Macro-F1.- 10-D geometry descriptor encodes spatial proximity (tip-vessel distance), motion (1st-/2nd-order changes, moving average), position (tip x, y), visibility (vessel area ratio, tip and vessel visibility), and topology (tip-inside-vessel). (ii) Reliability terms. Two reliability signals are estimated before fusion: a_t^vis measures geometric observation stability and visibility from g_t, while a_t^align=[1+cos(h_t^f,h_t^g)]/2 measures consistency between the visual and geometric latent representations. (Fig.2 notation will be aligned with the text formula.) (iii) Gating. ψ_t weights transient-memory readout, r_t (Eq.3) controls prototypal admission/compensation, and λ_t (Eq.5) balances transient and prototypal fusion. (iv) Hyperparameters. Transient/prototypal memory sizes (8,20) perform best among 2–40, with nearby settings within 0.79 Macro-F1.Threshold τ=0.50 performs best over 0.3–0.7, and fusion weight λ_t (Eq.5) is learned end-to-end, with category-adaptive values shown in Fig.3(b).
4.Segmentation performance, impact & instrument tip (R1, R2): Segmentation noise has limited impact due to our reliability-aware mechanisms. Vessel segmentation achieves Dice/IoU=84.64/73.37, and the visible active-tip center is manually annotated to isolate vessel-mask noise from tip-localization errors. Fig.3(a) analyzes visually unstable clips with inconsistent masks: F1 drops by 8.17 for our model vs. 32.84 for the baseline, showing that robustness arises from the proposed mechanisms rather than clean input. For prolonged unreliable spans, r_t blocks prototypal admission, preserving accumulated prototypes.
5.Failure analysis & clinical interpretation (R3): Errors mainly arise in non-routine motion patterns and concentrate more on the ambiguous approaching class than on the safety-critical touching class, as shown in Tab.1 (F1=64.3 vs. 73.7). For clinical use, r_t filters low-reliability evidence, while sustained high-risk predictions support surgeon verification.
6.Code & data release (Meta, R1, R2): We will open-source code, weights, and annotation protocol. The dataset will be released under a data-use agreement.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors present an interesting method for endoscopy. The reproducibility of the methodology should be better guaranteed in the manuscript. More details on the dataset should be provided.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The authors managed to opportunely improve the methods and the dataset description.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The responses addressed the concerns about description of the annotation process and reproducibility, missing algorithmic detail. Need to address the issues of robustness claims, descriptions of essential methodological components in final revision.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This paper introduces a novel and clinically meaningful task: vessel–instrument interaction anticipation in endoscopic submucosal dissection (ESD), shifting from retrospective recognition to proactive forecasting, supported by a new annotated multi-case dataset and a reliability-aware framework combining cross-modal latent recovery and dual asynchronous memory. R3 regarded the task formulation as a highly meaningful contribution and the architecture as technically impressive. R1 moved from Weak Reject to Accept post-rebuttal, finding the main concerns — annotation protocol and inter-annotator agreement, dataset splits, segmentation quality, and evidence that robustness stems from the proposed mechanisms — substantially addressed. R2 maintained reject on clarity grounds, which the authors should take seriously.
On balance, the novel task, clear clinical motivation, dataset contribution, and a convincing rebuttal support acceptance. For camera-ready, authors should fold in the rebuttal clarifications, deliver the committed code/data release, and discuss the lack of external validation.
