List of Papers Browse by Subject Areas Author List
Abstract
Text-to-video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety critical events may not follow the common structure, to unlock its full potential text-to-video retrieval must be able to handle implicit queries that require reasoning to identify the right videos (e.g., “the step right before clipping”). However, existing methods rely on global embeddings that cannot reason over such queries. We propose OR³, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR³ performs imagination-based retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures. OR³ achieves 57.6% R@1 and 77.3% R@5, outperforming the strongest baseline. These results demonstrate that OR³ enables fine-grained discrimination between visually similar OR video clips through temporal action reasoning. Code and dataset are available at https://anonymous.4open.science/r/OR3-FAEC/.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3993_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{SheYiq_Reasoning_MICCAI2026,
author = { Shen, Yiqing AND Ding, Hao AND Unberath, Mathias},
title = { { Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16893},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper studies reasoning text-to-video retrieval for operating room (OR) clips, with a focus on implicit queries that cannot be answered by simple visual matching, such as queries involving temporal, causal, procedural, or role-based reasoning. The authors propose a retrieval framework that presents each clip as an action-driven digital twin (ActDT) compared of temporally structured subject-action-object triplets. In addition, the authors constructs a benchmark from the MM-OR dataset, experimental results show that the proposed approach achieves superior performance than baselines.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The paper introduces imagination-based retrieval, where an LLM converts the input query into a hypothetical ActDT, so that retrieval becomes an intra-modal matching problem between imagined and actual ActDTs. (2) The authors define four categories of implicit OR retrieval queries and build a dataset, which is a meaningful step toward evaluating more realistic retrieval settings in operating-room videos. (3) The paper is well organized and easy to follow. And the experimental results are thorough and detailed.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
(1) The benchmark is promising but currently too limited. The experiments are conducted on 17 recordings, 386 clips, and 276 queries from robotic knee procedures. The dataset is relatively small for a retrieval framework that is presented as a general solution for OR reasoning retrieval. (2) The method’s novelty is meaningful but still somewhat increamental relative to very recent digital-twin-based reasoning retrieval work Reason T2V. (3) Although the large improvements on performance in Table 1, I think this is because the method benefiting from procedure-specific regularities and benchmark-specific patterns. It is unclear whether the method can generalize to larger datasets or different procedures. (4) The paper does not analyze inference efficiency, even though the proposed retrieval pipeline appears substantially more expensive than previous retrieval methods.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
I agree that reasoning-based retrieval for operating-room videos is an important and underexplored problem. The paper has several positive aspects. However, my concern is that the evaluation is conducted on a relatively limited benchmark. Although the improvements reported in Table 1 are large, they may partly reflect procedure-specific regularities and benchmark-specific patterns, rather than the robustness of the proposed retrieval framework itself. In addition, while the method is interesting, its novelty appears somewhat incremental relative to the very recent ReasonT2V line of digital-twin-based reasoning retrieval, with the main difference being the shift from object-centric to action-driven representations and the imagination-based matching strategy. Finally, the paper does not report inference efficiency, even though the multi-stage LLM-based pipeline with iterative refinement appears substantially more expensive than previous retrieval methods, which limits the assessment of its practical value.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I think this is a promising paper with a good problem setting and some interesting ideas, but the current version still falls short in terms of practical efficiency analysis, and strength of novelty relative to closely related recent work. For these reasons, I currently lean toward Reject.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors’ response has addressed my main concerns, I am willing to update my recommendation to accept the paper.
Review #2
- Please describe the contribution of the paper
This paper proposes OR3, a text-to-video reasoning retrieval framework for operating-room clips. The main methodological contribution is an action-driven digital twin representation that captures subject-action-object interactions over temporal intervals, together with an imagination-based retrieval strategy that generates a hypothetical digital twin from the query and performs intra-modal matching with a shared text encoder. The paper also introduces an evidence-grounded refinement step and a benchmark derived from MM-OR for implicit reasoning-based retrieval queries across temporal, causal, procedural, and role-based categories.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses a meaningful retrieval problem with clear practical motivation: users often need to retrieve OR clips through implicit descriptions that depend on temporal, causal, or workflow reasoning rather than only explicit visual content.
2.The proposed action-driven digital twin representation is well aligned with this problem because it encodes subject-action-object relations over time, which is more suitable than purely object-centric descriptions for distinguishing visually similar clips.
3.The imagination-based retrieval formulation is conceptually elegant. By generating a hypothetical structured representation from the query and matching it against actual clip representations with a shared text encoder, the method reduces the cross-modal alignment gap.
4.The additional analyses on refinement, candidate-pool size, and corpus scaling are useful and go beyond a single headline result.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The benchmark is relatively small and limited to robotic knee procedures, so the broader claim of operating-room reasoning retrieval is stronger than the current evidence supports.
2.Several implementation details are missing, especially for the LLM-based generation and refinement steps. Prompting, decoding settings, metadata usage, and generation robustness are not documented well enough for reproducibility.
3.The paper relies on multiple powerful external models and repeated LLM refinement, but runtime and computational cost are not reported, which is an important omission for a practical retrieval pipeline.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I find the problem relevant and the core idea interesting, especially the shift from object-centric to action-driven digital twins and the imagination-based retrieval formulation. The reported empirical gains are also impressive.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper introduces OR3, a reasoning-based text-to-video retrieval framework for operating room (OR) videos. The key contribution lies in representing video clips as action-driven digital twins (ActDTs), which encode subject–action–object interactions across temporal intervals, enabling fine-grained modeling of procedural dynamics
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A major strength is the introduction of action-driven digital twins (ActDTs). Unlike prior object-centric representations, ActDTs explicitly encode temporal action dynamics, which is crucial in surgical environments where visually similar frames differ only in interactions and procedural context
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The work builds upon existing ideas such as digital twin representations and LLM-based reasoning retrieval (e.g., ReasonT2V), with the main extension being the shift to action-driven representations and intra-modal matching. While effective, this does not constitute a fundamentally new paradigm. 2.the evaluation is limited to a single dataset and procedure type (robotic knee surgery). It remains unclear how well the method generalizes to: 1) Other surgical procedures 2) Different modalities. 3) Larger-scale datasets
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a strong application-driven contribution with clear empirical improvements and a well-designed experimental framework. The introduction of action-driven digital twins and imagination-based retrieval is interesting and effective.
Given the competitive standards, the paper falls slightly below the acceptance threshold, though it has clear potential with further refinement.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the reviewers for their constructive feedback. We address concerns below.
1.Benchmark Scale and Generalizability (R1, R2, R3, Meta): MM-OR is currently the only public OR dataset providing both scene graph annotations and phase/step segmentation needed to construct ActDTs and implicit queries. Our benchmark of 276 queries over 386 clips covers four reasoning categories with queries authored and cross-verified by two domain-expert annotators. OR3 uses no procedure-specific rules: the ActDT vocabulary, imagination prompts, and refinement logic are derived from generic OR metadata. To test generalizability across procedures (R1, R3), we applied OR3 trained on knee surgery to 48 clips from laparoscopic cholecystectomy in CholecT50, adapting only the entity and action vocabularies. OR3 achieves 41.3% R@1 on 32 temporal/procedural queries vs. 18.8% for ReasonT2V, confirming cross-procedure transfer. Regarding different modalities (R3), ActDT construction relies on RGB frames from room-mounted cameras, which is the standard capture modality across OR datasets (e.g., MM-OR, 4D-OR, CholecT50). Extending to endoscopic views requires only swapping the VLM and segmentation models, as the ActDT structure and downstream retrieval pipeline remain unchanged. We will include these discussions in the revision.
2.Novelty Relative to ReasonT2V (R1, R3, Meta): OR3 differs from ReasonT2V in three orthogonal dimensions, each validated by ablation (Tab.2). First, ActDT replaces per-frame object listings with temporally structured action primitives grouping concurrent interactions under non-overlapping intervals. Removing ActDT causes 15.2% R@1 drop, exceeding the 9.0% gap between ReasonT2V and the best embedding baseline. Second, imagination-based retrieval is a new paradigm with no counterpart in ReasonT2V: an LLM generates a hypothetical ActDT so retrieval becomes intra-modal matching with a single encoder, eliminating paired cross-modal encoders (12.1% contribution). Third, evidence-grounded refinement revises the imagined representation itself rather than invoking specialist models to fill gaps (7.6% contribution). These are distinct design choices whose combined effect yields 33.4 R@1 points over ReasonT2V.
3.Inference Efficiency (R1, R2, Meta): We provide runtime analysis per query, averaged over the test set on a single RTX 4090.ActDT construction is offline and one-time: 38.2s per clip (VLM 24.1s, SAM-3 9.6s, depth 4.5s). At query time, imagination takes 1.8s (single LLM call), encoding and matching takes 0.12s, and refinement averages 4.2s (1.6 rounds before early stopping). Total online latency is 6.1s per query over 386 clips. ReasonT2V requires 8.4s due to two-encoder matching and multi-step LLM reasoning with just-in-time specialist invocations. Embedding methods are faster (<0.1s) but achieve below 16% R@1.We will add a runtime table in the revision.
4.Implementation Details and Reproducibility (R2, Meta): We will release full prompt templates, decoding configurations (temperature=0.3, top-p=0.95 for imagination; temperature=0.1 for refinement), and metadata specifications in the supplementary. The anonymized code repository (noted by R1, R3) contains all prompts and reproducible training scripts. We will also document generation robustness: across 5 runs, the LLM produces valid ActDT JSON in 97.4% of imagination calls, with malformed outputs caught by schema validation and re-prompted automatically.
5.Performance vs. Benchmark-Specific Patterns (R1): Three observations argue against benchmark overfitting. First, the scalability analysis shows R@1 remains stable (57.6% to 55.8%) when expanding the candidate pool from 96 to 386 clips with held-out recordings, indicating no corpus-specific shortcuts. Second, all results are averaged over five seeds with reported standard deviations. Third, the cross-procedure transfer result (Point 1) confirms OR3 maintains its advantage on unseen procedures.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a reasoning-based text-to-video retrieval framework for operating room clips that represents each clip as a so-called action-driven digital twin, performs retrieval via LLM-generated hypothetical representations, and applies evidence-grounded refinement.
Reviewers comment positively on the practical motivation for handling reasoning queries, find the action-driven representation well-aligned with the task, the imagination-based matching as conceptually elegant, and value the new benchmark.
Concerns recur around a small benchmark which weakens generality, incremental novelty relative to prior reasoning retrieval work, missing implementation details that limit reproducibility, and the absence of inference efficiency or computational cost analysis, etc.
Reviews are mixed and the authors may respond to the concerns in the rebuttal.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After the rebuttal, the majority of reviewers recommend acceptance.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Based on my reading of the paper and the reviews provided my recommendation is for the paper to be accepted. The reasons is the paper addressed an important and interesting problem with adequate methodological novelty and introducing a new benchmark. The authors also did a good job with the rebuttal.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The paper presents an interesting action-driven digital twin formulation for text-to-video retrieval in operating-room clips, with clear application motivation and encouraging empirical gains. However, the review scores remain borderline overall. Since some concerns about refinement and strength of contribution remain insufficiently resolved, the paper is slightly below the acceptance threshold.