List of Papers Browse by Subject Areas Author List
Abstract
While traditional surgical action triplet detection has achieved remarkable progress, these closed-set models often fail in highly variable, real-world clinical environments where completely novel instruments, anatomical structures, and actions frequently emerge. To address these limitations, we introduce SurgZSD, a novel framework that leverages medical priors to facilitate attribute composition and joint inference for surgical triplets. Specifically, we initially fine-tune Grounding DINO to establish the precise spatial grounding of the surgical instruments. Following this localization stage, we introduce the Medical Knowledge-Guided Attribute Composition (MKAC) module. Guided by medical priors, this module leverages a pretrained Vision-Language Model (VLM) to parse the localized bounding box regions into structured semantic attributes, thereby composing unseen surgical instruments, targets, and verbs. Subsequently, to accomplish comprehensive surgical triplet recognition, we introduce the Clinical Feasibility-Constrained Joint Inference (CFCJI) module. This module introduces clinical feasibility matrices constructed based on medical priors. By formulating a Top-$K$ joint inference objective function, it globally integrates the visual parsing scores with a clinical compatibility penalty term to yield the final surgical action triplets. Extensive experiments demonstrate that SurgZSD not only achieves state-of-the-art performance in zero-shot scenarios but also retains competitive performance under closed-set settings.The source code is available at https://github.com/DengNova/SurgZSD.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3187_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/DengNova/SurgZSD
Link to the Dataset(s)
N/A
BibTex
@InProceedings{DenZux_SurgZSD_MICCAI2026,
author = { Deng, Zuxing AND Li, Ling AND Li, Xiaojian AND Liu, Yuting AND Ma, Xiang AND Zhou, Wenrui},
title = { { SurgZSD: Zero-Shot Surgical Action Triplet Detection via Attribute Composition and Clinical Feasibility Inference } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper addresses the problem of zero-shot detection of surgical action triples and proposes a framework called SurgZSD. The proposed framework introduces a Medically Knowledge-Guided Attribute Combination (MKAC) module, which decomposes entities into structured attributes and performs combinatorial reasoning using a visual-language model. Additionally, it includes a Clinical Feasibility Constraint Joint Inference (CFCJI) module, which integrates prior medical knowledge to enforce compatibility constraints among the elements of the triples. Experiments demonstrate that the method achieves good performance in zero-shot scenarios.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.This paper addresses a significant and practical challenge in surgical scene understanding—the ability to generalize to unseen instruments, targets, and actions—which is of great importance for real-world applications in various surgical procedures.
2.Rather than performing direct classification, the method utilizes VLMs to extract structured attribute representations, thereby enabling compositional generalization.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.To my knowledge, both the ITG-Trip and MCIT-IG methods pre-trained the object detection model Deformable DETR on Cholec80.In contrast, this paper fine-tunes Grounding DINO on the CholecTrack20 dataset. The authors did not conduct ablation experiments for Stage A, which raises doubts as to whether the model’s performance gains are primarily attributable to Stage A.
2.This method does not perform training or inference at the frame level, thereby neglecting the temporal dependencies that are crucial for understanding surgical actions—a factor that is critical for contextual inference in surgical procedures.
3.The proposed framework consists of decoupled components across multiple stages, which limits end-to-end optimization and hinders clinical deployment.
4.As shown in Figure 2, after Stage 2, the instrument has already obtained a unique predicted probability. Why is it necessary to perform a Top-K operation on the instrument in Stage 3?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper addresses the challenge of zero-shot surgical action triplet detection and demonstrates a degree of innovation. However, I have serious concerns regarding the fairness of the comparisons and the model’s effectiveness.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The author has addressed some of my concerns, but I remain worried about the fairness of the Stage A and the clinical applicability of this model.
Review #2
- Please describe the contribution of the paper
(1) The paper introduces SurgZSD, the first framework proposed for zero-shot surgical action triplet detection. (2) It proposes the Medical Knowledge-Guided Attribute Composition (MKAC) module, which uses a Vision-Language Model to parse localized regions into structured attributes based on medical prototypes to recognize unseen entities. (3) It introduces the Clinical Feasibility-Constrained Joint Inference (CFCJI) module, which formulates a joint inference objective constrained by clinical feasibility matrices to globally integrate visual scores and output coherent surgical action triplets.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) This paper formulates a novel and challenging problem by introducing a strict zero-shot paradigm to surgical scene understanding, where training and testing classes are completely disjoint. (2)The framework innovatively synergizes an open-set object detector with a VLM and an LLM to bridge the domain gap without requiring fully annotated fine-grained surgical data for novel classes.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
(1) The topic shares conceptual similarities with a recent ICLR submission, “Cognitive-Awakening Chain-of-Surgery for Compositional Zero-Shot Surgical Triplet Recognition.” Although that work focuses on triplet recognition rather than detection, the underlying compositional zero-shot motivation is highly related. The authors should cite and explicitly discuss the differences and relative advantages of their proposed framework compared to this work. (2) The framework currently infers actions based purely on spatial interactions within isolated frames. Since surgical actions are inherently continuous and dynamic, have the authors considered incorporating temporal information in VLMs to further enhance triplet detection accuracy? (3) To ensure strict reproducibility, the promised code release must guarantee that all dependencies, the exact structured prompt templates used for the VLM, and the fully generated Knowledge Base are open-sourced alongside clear, step-by-step documentation.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a highly original and clinically relevant approach to surgical scene understanding by addressing the inevitable presence of novel instruments and anatomies in real-world deployments. The proposed integration of medical prior knowledge for attribute composition and clinical feasibility inference is logically sound and well-justified. Furthermore, the zero-shot evaluation on 1377 manually annotated images clearly demonstrates that SurgZSD significantly outperforms baseline models, which struggle with spatial localization or domain gaps. Although the proposed method is quite straightforward, I find it to be highly simple and effective.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I maintain my original opinion based on the concerns and evaluation provided in my previous review.
Review #3
- Please describe the contribution of the paper
The paper proposes a framework for zero detection of surgical action triplets from images with the help of object detectors, VLM for describing the instruments and then a clinically constructed knowledge base to identify the instruments based on description followed by identification of actions and anatomy.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper is well presented and structured, stating clearly the focus on each component of the method.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The VLM questionnaire for action and anatomy are not clearly stated in the paper. Fig 2 could be modified to show the questions for each of I,V,T. 2.A curious question is does 3 questions with 4 options each sufficient to represent all laparoscopic surgical tools? Any reference / proof to support this i.e., some kind of graph or visual representation showing the known surgical tools to mapping to the questions and options in one-on-one manner. This kind of representation would strengthen the completeness of the proposed method. 3.What data is the VLM pretrained on? Is it a general purpose VLM or surgically trained VLM? If it is general purpose VLM then, in the extended version of this work it would be better to show an ablation with different off the shelf VLMs such that the effectiveness of the framework is portrayed. 4.Does the knowledge base contain information generic to all surgeries and triplets? If it contains information related to the triplets in evaluation then the zero-shot concept applies only to detecting the instruments and the title need modifications accordingly. 5.Since Dino is one method, if and when an extended version of this work is taken up different zero-shot detection methods need to be considered for ablation.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The concept of the work is novel, however there are few aspects which need further clarifications and the paper could be accepted if those issues are addressed.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the reviewers and AC for constructive comments and recognition of novelty and clinical relevance. Our responses are provided below. 1.Fairness and Stage A (R1, R3, AC). We understand the reviewer’s concern about fairness. The comparison with ITG-Trip and MCIT-IG is intended to show that a zero-shot-oriented framework can still retain reasonable capability under the closed-set setting. CholecTrack20 is derived from Cholec80 and provides instrument localization, making it a reasonable source for detector fine-tuning. Deformable DETR is a closed-set detector, whereas the zero-shot setting may contain unseen instruments, thus Grounding DINO is used as an open-vocabulary model. SurgZSD is detector-agnostic: any reliable open-set instrument detector can serve as Stage A. We agree that detector-level ablations would strengthen the analysis and will include them as future work. Stage A provides localization proposals, while the main methodological contribution lies in the MKAC and CFCJI modules. The ablation studies in the manuscript demonstrate the effectiveness of these two modules. 2.VLM attribute-query scheme (R3, AC). We apologize that Fig. 2 simplified the query design and caused misunderstanding. “3 questions with 4 options” is only a compact illustrative example, not the full attribute vocabulary. In our work, the attribute dimensions are fixed for interpretability, but the candidate options are expandable. Thus, the method does not assume that four options can cover all instruments. A new entity can be added by defining its attribute prototype in the attribute space, without additional visual samples. We will revise Fig. 2 and provide the prompts and attribute vocabularies. 3.Temporal information, optimization, and deployment (R1, R2). We agree temporal context is important, especially for verb recognition. This work focuses on frame-level zero-shot triplet detection, temporal information is complementary to the framework. A natural extension is to replace the frame-level VLM parser with a video-aware parser that can incorporate motion and temporal context. This will be considered as future work. SurgZSD is intentionally modular: the detector, VLM attribute parser, and feasibility inference layer can each be independently inspected and optimized. Although this sacrifices part of the end-to-end optimization capability, it improves the transparency required for clinical review. 4.Top-K operation in Stage C (R1). Although Stage B scores instrument candidates, the top-1 prediction is only used for the instrument detection comparison experiment. For complete triplet prediction, relying solely on the top-1 prediction may be unreliable, because unseen instruments can share similar attributes. Stage C is designed to avoid locally confident but clinically implausible combinations. Therefore, Top-K serves global triplet optimization, rather than redundant prediction. We will clarify this in the revision. 5.Knowledge base and VLM usage (R3). The knowledge base contains no triplet-level information, such as triplet labels or frequency statistics, avoiding data leakage. It is built separately for instruments, targets, and verbs using general medical prior knowledge and semantic attribute prototypes generated with GPT-5.2.The VLM reasoning module in Stage B and Stage C is Qwen3VL-Plus, a general purpose VLM without surgical-domain training or fine-tuning. Thank you for the valuable suggestion. We will evaluate different off-the-shelf VLMs and further explore fine-tuning domain-specific surgical VLMs in the extended version. 6.CoCoS Comparison (R2). First, CoCoS recognizes unseen combinations of seen components, whereas SurgZSD detects unseen instruments, targets, and verbs. Next, SurgZSD relies on medical attributes and clinical feasibility constraints, while CoCoS leverages temporal Glance-Gaze-Think reasoning. Thus, they are complementary. 7.Reproducibility (R2). Upon acceptance, we will immediately release the code.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
SurgZSD presents an approach to zero-shot surgical triplet detection combining open-set Grounding DINO with a Medical-Knowledge-Guided Attribute Composition module and a Clinical Feasibility Constraint. Two key issues require response before a final decision: (1) Fair-comparison concern - prior triplet-detection baselines (ITG-Trip, MCIT-IG) pretrain Deformable DETR on Cholec80 training data, while this work fine-tunes Grounding DINO on CholecTrack20 (2) Scope of the VLM attribute-query scheme — the 3-question/4-option protocol has not been validated to cover the full instrument space. The authors should address these issues thoroughly in the rebuttal.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After reviewing the rebuttal letter and reviewers’ post-rebuttal comments, I recommend acceptance.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
All reviewers were satistfied with the rebuttal. The authors should address the remaining concerns in the camera-ready version.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The rebuttal clarifies key points such as the detector setup, attribute-query scheme, and design choices, but does not fully resolve concerns about fair comparison, coverage of the attribute space, and lack of temporal modelling. Overall, the reviewers in general lean toward acceptance due to the novelty of the problem and approach, but evaluation limitations remain.
