List of Papers Browse by Subject Areas Author List
Abstract
Unified surgical video understanding aims to handle phase recognition, CVS assessment, and fine-grained action triplet recognition within a single framework, yet remains challenged by highly long-tailed surgical category distributions. Existing VLM-based approaches encoded category knowledge parametrically, causing rare but clinically critical triplets to be under-learned due to frequent-class dominance and fragile cross-modal alignment. To address this, we propose Retrieval-Augmented Surgical Perception (RASP), which reduces the reliance on model parameters for category knowledge by using an external memory bank that exploits standardized surgical workflows. RASP includes a Contextual Representation and Semantic Retrieval (CRSR) module that compresses temporal windows into context embeddings to recall coarse-level semantic priors and fuse them with current-frame observations. It further employs a Multi-Context Feature Representation (MCFR) module with evidence-aware cross-attention for fine-grained prior selection and an evidence gate for fallback when retrieval is uninformative. Extensive experiments show that RASP consistently outperforms state-of-the-art methods, improving AP-IVT on rare triplets by 6.3% while achieving superior multi-task performance. The code is available at https://github.com/yuxiangluo-ops/RASP.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4703_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/yuxiangluo-ops/RASP
Link to the Dataset(s)
N/A
BibTex
@InProceedings{LuoYux_RASP_MICCAI2026,
author = { Luo, Yuxiang AND Xu, Qing AND Ouyang, Yuqi AND Chen, Zhen},
title = { { RASP: Bridging the Long-Tail Gap in Surgical Video Understanding via Retrieval-Augmented Perception } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose RASP (Retrieval-Augmented Surgical Perception), a novel framework designed to mitigate the long-tailed distribution problem inherent in surgical video understanding. Recognizing that standard parametric Vision-Language Models (VLMs) often fail on rare classes due to gradient dominance by frequent categories, RASP introduces a non-parametric retrieval mechanism. The approach utilizes a two-stage training pipeline: first, learning context-enhanced frame representations via a Temporal Context Encoder, and second, retrieving semantic priors from an external memory bank using visual-to-visual similarity. These retrieved priors are fused with local features using an evidence-aware gate. The method is evaluated on Cholec80, CholecT50, and Endoscapes2023, demonstrating significant improvements over multi-task baselines.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper tackles a critical issue in surgical AI—the long-tailed distribution of action triplets and adverse events. The motivation that parametric VLMs struggle with rare classes due to fragile cross-modal alignment is well-founded and clinically relevant. 2.Adapting retrieval-augmented mechanisms to surgical video perception, specifically using visual-temporal context as keys to bypass text alignment issues for rare classes, is a creative contribution. 3.The reported relative improvement of 73.2% on rare action triplets is substantial. This suggests the method effectively preserves knowledge about infrequent but safety-critical events, which is a key requirement for clinical deployment. 4.The framework is validated across three distinct tasks on multiple standard benchmarks, demonstrating robustness and versatility beyond a single metric.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The manuscript lacks quantitative analysis of inference latency or computational overhead. Given that retrieval complexity scales with the memory bank size, the real-time applicability of RASP in an intraoperative setting is unclear. 2.While compared against strong multi-task models (e. g. , MML-SurgAdapt), the paper does not include comparisons against simpler long-tail learning strategies (e. g. , Class-Balanced Loss, Focal Loss) applied to the same backbone. This makes it difficult to isolate whether the gains come from the retrieval mechanism or simply better handling of class imbalance. 3.The decision to freeze Stage I parameters during Stage II prevents the encoder from adapting to retrieval feedback. While this prevents representation drift, it may limit the overall optimization potential compared to end-to-end trainable alternatives.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.Please include a discussion or measurement of inference time (FPS) compared to parametric baselines. Even if the method is slower, quantifying the trade-off between accuracy and latency would strengthen the practical assessment. 2.If possible, include results for standard long-tail loss functions on the CLIP backbone. This would help confirm that the retrieval module provides value beyond standard re-weighting techniques. 3.Given the storage concerns, could the memory bank be compressed or clustered to reduce size without significant performance loss? A brief discussion on this would address deployment feasibility. 4.Please clarify why end-to-end training was not pursued. A brief explanation of the stability issues encountered during attempts to train jointly would justify the two-stage design choice. 5.While the results on cholecystectomy are strong, a brief discussion on how this might translate to more variable procedures would broaden the impact of the work.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper presents a compelling and novel approach to a persistent problem in surgical video understanding. The RASP framework offers a distinct perspective by leveraging non-parametric retrieval to handle long-tailed distributions, and the empirical results, particularly on rare action triplets, are impressive. The significant gains in safety-critical rare classes justify the interest in this method. However, the recommendation is conservative due to concerns regarding inference efficiency, storage requirements, and the lack of comparison against simpler long-tail baselines. These issues do not invalidate the core contribution but should be addressed to clarify the practical applicability of the method.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper presents a good idea of incorporating retrieval with a memory bank into a vision–language alignment framework (CLIP-like models) for surgical phase recognition, CVS classification and action triplet recognition. The experiments showed performance gains over other models, especially in tail cases.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper clearly highlights the issue of fragile cross-modal alignment in parametric models, especially for tail classes, which is an important and relevant problem.
- The design choices are well-motivated, particularly the use of temporal context embeddings and the gating mechanism for integrating retrieval features.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The paper emphasizes “fragile cross-modal alignment based on parametric learning” as a limitation and claims that “non-parametric retrieval mechanisms preserve tail-class knowledge independently of gradient-driven dynamics.” I am not fully convinced, as the memory bank is still constructed through a parametric process, which directly affects tail-class representations. This raises a related question: if the memory bank for tail classes is suboptimal, why does retrieval still improve performance? I would appreciate further clarification from the authors on this point.
- The paper does not mention whether the code will be made publicly available, which limits reproducibility.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- The paper could be strengthened with additional analysis of the retrieval process. For example, it would be helpful to quantify how much the model relies on the retrieval branch versus the parametric branch (e.g., via gating weights), and to include visualizations of the top-k retrieved samples to better understand what information is being leveraged.
- It would also be interesting to include an evaluation on a surgical-domain foundation model (e.g., a CLIP-like model trained on surgical data), as this could provide a more domain-relevant baseline and further strengthen the empirical validation.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposes a well-motivated retrieval-augmented framework for improving long-tail surgical video understanding, showing clear and consistent gains on tail classes. There could be more analysis on how the retrieval mechanism contributes to performance and how it interacts with the parametric model, which would improve clarity and understanding. However, given space constraints, this is understandable and could be addressed in future work.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper’s main contribution is RASP, a retrieval-augmented framework for unified surgical video understanding that addresses the long-tail weakness of purely parametric VLM-based methods by storing and retrieving category-level knowledge from an external memory bank instead of relying only on learned weights. This paper also introduces a coarse-to-fine retrieval pipeline: a contextual representation and semantic retrieval module uses visual temporal context to recall relevant historical priors, and a multi-context feature representation module refines these priors with evidence-aware cross-attention and gating.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper focuses on unified surgical video understanding across both coarse procedural phases and fine-grained action triplets, and it explicitly frames the long-tail issue as a limitation of purely parametric VLM-style learning, which is a practically relevant concern in surgical data. 2.The method has a fairly structured coarse-to-fine design. RASP first builds temporal-context queries for coarse retrieval, then uses a multi-context cross-attention module with an evidence gate to select fine-grained semantic priors, and the two-stage optimization is a sensible way to reduce representation drift after memory-bank construction. 3.There is some targeted evidence that the retrieval modules help the tail classes. In the ablation, the full model improves AP-IVT over the baseline and raises AP-IVTtail from 8.6 to 14.9, and the window-size / neighbor-count studies suggest the authors did some effort to validate the retrieval setup rather than reporting only one configuration.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The central long-tail claim is not fully supported by the evaluation. The paper’s motivation is that RASP specifically addresses long-tailed surgical distributions and preserves tail-class knowledge independently of gradient-driven learning, but the experiments provide only one explicit tail-focused metric, AP-IVT_tail on CholecT50.For the main benchmarks, the gains over the strongest multi-task baseline are otherwise fairly modest—e.g., AP-IVT improves from 31.6 to 33.9 and CVS mAP from 55.9 to 57.3—so it is difficult to conclude that the improvement truly comes from solving the long-tail problem rather than from added temporal modeling, retrieval capacity, or extra architectural complexity. 2.The retrieval mechanism appears only partially effective for the rare classes it is meant to help most. The paper emphasizes a 73.2% relative improvement on rare triplets, but the absolute value is still only 14.9 AP-IVT_tail, which remains far below the overall 33.9 AP-IVT and far below the easier marginal components such as AP-I/AP-V/AP-T. This suggests that retrieval may alleviate the tail problem to some extent, but it does not yet “bridge” the gap in a strong sense, and the practical recognition quality on rare triplets is still quite limited. 3.The paper provides limited evidence about whether the retrieved information is actually meaningful and reliable. Although the method relies heavily on visual retrieval, evidence-aware gating, and coarse-to-fine semantic prior selection, the analysis is almost entirely numerical: there are no qualitative retrieval examples, no inspection of retrieved neighbors, no analysis of when the gate suppresses bad evidence, and no failure-case discussion. Given that the method’s main claim is that visual retrieval can safely replace fragile cross-modal alignment for rare classes, this lack of retrieval-specific analysis makes the mechanism harder to trust.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please check the weakness.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
N/A
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The reviewers unanimously recommend a the acceptance of this paper, acknowledging the paper’s innovative use of Retrieval-Augmented Surgical Perception (RASP) to address the critical “long-tail” distribution problem in surgical video understanding. Reviewers highlighted the creativity of using a non-parametric memory bank to preserve knowledge of rare classes and the impressive 73.2% relative improvement in rare action triplets. While the methodology is well-structured and clinically relevant, final acceptance is contingent on the authors refining the manuscript to address three key areas: computational feasibility, comparative depth, and qualitative transparency. Specifically, the authors could include an analysis of inference latency (FPS) to justify real-time intraoperative potential, compare the framework against standard long-tail baselines (e.g., focal or class-balanced losses) to isolate the benefits of retrieval, and provide qualitative visualizations of retrieved neighbors and gating weights to demonstrate that the mechanism functions as intended. Finally, the authors should discuss the optimization trade-offs of their two-stage training design and the potential for memory bank compression for easier deployment.