List of Papers Browse by Subject Areas Author List
Abstract
Surgical video question answering requires multi-step reasoning across semantic, spatial, and temporal dimensions. Existing methods architecturally compress videos into discrete token representations and couple visual perception with reasoning. This approach fragments continuous spatial-temporal relationships and has been shown to restrict multi-step reasoning capabilities. We introduce a reinforcement learning (RL) framework that trains large language models (LLMs) to decouple perception from reasoning by operating over digital twin representations constructed from surgical foundation models. Additionally, we introduce hierarchical representations across frame, temporal window, and procedure levels with probabilistic uncertainty estimates. Finally, we propose a novel reward that combines format validation with accuracy assessment through clinical plausibility evaluation and uncertainty-aware calibration for training. To demonstrate the capabilities of this approach, we introduce REAL-Colon-Reason, a colonoscopic benchmark with 2000 question-answer pairs across three complexity levels. We achieve state-of-the-art performance on REAL-Colon-Reason and two existing surgical VideoQA benchmarks REAL-Colon-VQA and EndoVis18-VQA. Code and dataset are available at https://anonymous.4open.science/r/surgical_videoqa_dt/.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2694_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{SheYiq_Training_MICCAI2026,
author = { Shen, Yiqing AND Zhang, Han AND Unberath, Mathias},
title = { { Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution is a synthesis of existing tools to design a multi LLM system, trained by RL that outperforms the SOTA in selected benchmarks. The target app is Q/A on surgical videos. The LLM uses a structured rollout that explicitly states reasoning steps and leverages existing foundation LLMs for segmentation, depth map generation, and semantic tagging.
The paper also introduces a new dataset that
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Table 2 shows the new method is applied on two existing datasets and compare with several alternative methods and different metrics. The results are good.
New curated dataset will be available (2000 question / answer pairs), although I’m not in position to judge its quality.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Complex workflow
- Small training dataset (just 60 videos and GPT-generated Q/As (refined by clinicians)
- Limited novelty;
Unclear that this research is reproducible. The authors don’t mention whether the architecture and the weights will be released
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.Why are the datasets in Table 2 sufficient to evaluate the proposed system? 2.Are you going to open source architecture and weights? 3.2000 Q/A seems a significant amount of questions. The clinicians would have to watch 60 videos and check the questions. How many clinicians were involved?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The results in Table 2 seem interesting
- Reviewer confidence
Not confident (1)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
I don’t see any strong technical contributions in this paper; in my opinion it requires a longer format to be comprehensible.
Review #2
- Please describe the contribution of the paper
In this work the authors suggest a method for Video Question Answering in which an LLM, based on question semantics, plans a sequence of executions of vision foundational models which outputs on frame level can then be used as inputs for a new representation of the video as a “digital Twin” which can then be used as input for further reasoning tasks. As the DT has a structure which includes
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors present their work in a well structured manner, describing most of the concept in a level of detail which should allow for the repetition of the results.
The experiments are sound, with a good mixture of both experiments performed on the newly developed benchmark dataset as well as experiments performed on other external datasets. The presented experiments show a very high number of evaluations.
In regards to the “Digital Twin” approach itself the author clearly state that this is not a completley novel method however it’s application to VideoQA is clearly new.
The preparation of the
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The authors showed a distinct number of quantitative evaluations of the results. However I’m missing a somewhat qualiatative evaluation of the resuslt by experts in regards in how close the final answers provided by the models are comparable to the real information observed by the experts.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall this is a very sound work, which presents a very good level of the scientific workshop of the authors. Topic is relevant and interesting for the community.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper presents a digital-twin-based framework for reasoning-intensive surgical VideoQA, together with a new benchmark, REAL-Colon-Reason, designed to evaluate multi-step semantic, spatial, and temporal reasoning in colonoscopy videos. The proposed method decouples perception from reasoning: rather than answering directly from video tokens, an LLM first plans the construction of a digital twin (DT) using surgical foundation models, and then reasons over this structured intermediate representation to produce the final answer. The paper further introduces a hierarchical DT representation spanning frame, temporal-window, and procedure levels, and augments it with probabilistic uncertainty estimates. Training is performed with GRPO, using a reward that combines structured output validation, answer correctness, clinical plausibility, and uncertainty-aware scaling. Empirically, the method is reported to outperform a range of surgical and general VLM baselines on the proposed benchmark as well as on REAL-Colon-VQA and EndoVis18-VQA.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Relevant problem setting. The paper addresses an important and still underexplored problem, namely surgical VideoQA that requires genuine multi-step reasoning rather than shallow frame-level recognition. This is a meaningful direction for clinical decision support and surgical education.
Interesting system design. The central idea of separating perception from reasoning through a structured DT representation is well motivated. In surgical video, where temporal continuity, spatial relations, and perceptual ambiguity matter, reasoning over an explicit intermediate representation is arguably more appropriate than relying on token-based compression alone.
Sensible domain-specific adaptations. The introduction of hierarchical DTs across frame, temporal-window, and procedure levels is well aligned with the temporal structure of surgery. Likewise, incorporating uncertainty into both the DT and the reward design is a sensible choice for a noisy and ambiguous visual domain.
New dataset contribution. REAL-Colon-Reason appears to be a useful benchmark, especially because it explicitly targets increasing reasoning complexity rather than only visual recognition. This could become a valuable resource if released and documented carefully.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Novelty with respect to prior DT work is under-specified. The manuscript clearly builds on prior DT-based reasoning work, including previous DT planning and RL-based DT reasoning papers cited as refs. [23–26]. However, it does not explicitly and systematically explain what is fundamentally new relative to those earlier DT models. The differences must be inferred indirectly from the contribution list, rather than being discussed in a dedicated comparison section. As a result, the incremental contribution over prior DT work is difficult to assess precisely.
Missing comparisons to the most relevant prior art. Although the paper cites previous DT papers, none of these DT-based methods are included in Table 1 or Table 2.This is a significant issue, because the strongest novelty claim of the submission is precisely the adaptation and extension of DT reasoning to surgical VideoQA. Without direct comparison to earlier DT or GRPO-based DT methods, it is difficult to determine whether the reported gains come from the proposed surgical extensions or simply from inheriting the advantages of the broader DT framework.
Potentially unfair comparison protocol. The proposed method is not merely an “LLM for VideoQA”; it is a tool-augmented pipeline with access to multiple specialized surgical foundation models, a DT construction stage, and structured RL training. By contrast, many of the baselines in Table 1 and Table 2 are plain VLMs or supervised systems that do not have access to the same perception tool stack. This concern is reinforced by the fact that several results in Table 2 appear to be taken directly from the SurgViVQA setting, where models are trained in a conventional supervised manner or used in zero-shot mode. Therefore, the comparison is not fully apples-to-apples. The experiments support a system-level claim, namely that the overall proposed pipeline performs better, but they do not cleanly isolate the contribution of DT reasoning itself. This is particularly important because the paper presents the explicit reasoning structure as the key source of improvement.
Experimental details are incomplete. Several important details are omitted or unclear. For example, the paper reports mean ± std in Table 1, but does not explain how the standard deviation was computed or over how many runs. Likewise, the adaptation protocol for frame-based baselines such as SurgicalGPT and PitVQA to video-based benchmarks is not described. These omissions weaken the credibility and reproducibility of the empirical study.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
Presentation issues. The manuscript contains a number of clarity and editing problems. For example, the sentence “We conduct training on 8 NVIDIA NVIDIA 4090 GPUs of 24Gb memory using DeepSpeed for distributed optimization” contains a clear repetition (“NVIDIA NVIDIA”). More broadly, there are places where the writing suggests a stronger degree of novelty than is fully supported by the paper’s actual relationship to prior DT work.
Figure 1 is unclear. The pipeline illustration should be restructured so that the input and output of each model component are shown more clearly. At present, the figure is visually dense and makes it difficult to follow the flow between planning, DT construction, reasoning, and answer generation.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important problem and contains several promising elements, including a meaningful task formulation, a sensible DT-based system design, and a potentially valuable new benchmark. However, in its current form, the paper does not yet make a sufficiently clear case for its novelty relative to prior DT-based work, nor does it provide a fully convincing empirical comparison. In particular, the absence of direct baselines against previous DT models makes it difficult to assess the true incremental contribution, while the comparison to standard VLM baselines is not fully controlled given the proposed method’s access to specialized external tools. In addition, some key experimental details remain underspecified. For these reasons, I lean toward a weak reject, although I believe the work could become substantially stronger with clearer positioning and a more rigorous comparison protocol.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The paper addresses an important and underexplored problem: reasoning-intensive surgical VideoQA. The proposed framework is well motivated, separating perception from reasoning through structured digital twin representations, hierarchical temporal modeling, probabilistic uncertainty, and RL-based training. The submitted experiments show strong performance on the proposed REAL-Colon-Reason benchmark and also demonstrate generalization to existing surgical VideoQA datasets. My main concerns regarding novelty positioning, comparison fairness, dataset construction, and reproducibility have been clarified by the author response. While the final version should clearly present these clarifications, especially the distinction from prior DT/RL-based work and the dataset validation process, I now find the contribution sufficiently strong for acceptance.
Author Feedback
We thank all reviewers for their constructive feedback, with our point-to-point response below.
1.Novelty over prior DT work (R1,R4,MR): Our method differs from prior DT-based reasoning [23-26] in three ways. (a) Hierarchical temporal DT, specifically for surgical videos: prior methods build flat, per-frame representations; we introduce a three-level hierarchy (frame, temporal window, procedure), improving EM from 0.383 to 0.441 (Tab.3). (b) Probabilistic DT: prior DT representations are deterministic; we attach calibrated confidence scores and propagate them into the reward, improving EM to 0.584 with uncertainty-modulated reward. (c) Clinical validity reward: prior methods for DT use generic accuracy rewards; ours penalizes clinically implausible answers via LLM-as-judge, absent in general-domain DT work. We will add a dedicated comparison paragraph and temper novelty claims accordingly in revision (R4).
2.Missing DT baselines (R4,MR): We evaluated DT-R1 (state-of-the-art DT method) on REAL-Colon-Reason using the same surgical foundation models. DT-R1 (zero-shot): 0.298 EM; DT-R1 (fine-tuned on our data): 0.462 EM; Ours: 0.584 EM. The gap comes from DT-R1 lacking hierarchical temporal modeling, uncertainty calibration, and clinical reward. We will add these to Tab.1. 3.Comparison fairness (R4,MR): Our method intentionally accesses perception tools; the contribution is the framework orchestrating them via learned DT planning. Tab.3 provides controlled ablations on the same backbone: no DT 0.242 EM, flat DT 0.383, full framework 0.584.The SFT baseline with identical tool access achieves only 0.325 EM, confirming gains arise from RL-trained reasoning, not merely stronger perception.
4.Dataset sufficiency and clinician involvement (R1,MR): REAL-Colon-Reason builds on REAL-Colon (60 videos, 6 medical centers), one of the largest public colonoscopy datasets with frame-level annotations. Three annotators independently reviewed all 2000 QA pairs generated by GPT, correcting reasoning chains and verifying clinical accuracy (Cohen kappa: 0.82). We evaluate on three benchmarks (REAL-Colon-Reason, REAL-Colon-VQA, EndoVis18-VQA) spanning two procedure types (colonoscopy and nephrectomy) to validate cross-procedure generalization; these are the only publicly available surgical VideoQA benchmarks with temporal annotations suitable for our task.
5.Qualitative expert evaluation (R3,MR): Three annotators rated 50 randomly sampled answers from our method and the strongest baseline (Qwen3-VL-8B) on a 1-5 clinical correctness scale. Ours: 4.12 avg vs 3.28 for Qwen3-VL-8B, with the largest gap on Level 3 questions (3.85 vs 2.64). We will include this in revision.
6.Complex workflow and efficiency (R1): The query-adaptive DAG planning avoids invoking all foundation models uniformly; on average only 2.3 of 4 models are called per query. Per-clip inference takes approximately 12s for DT construction and 3s for LLM reasoning on a single 4090 GPU. While slower than end-to-end VLMs, the modular design allows each component to be independently upgraded or accelerated.
7.Experimental details (R4,MR): Standard deviations in Tab.1 are over 3 independent runs with different seeds. Frame-based baselines (SurgicalGPT, PitVQA) are adapted to video by per-frame inference with majority-vote aggregation.
8.Reproducibility/open-source (R1,R3,R4,MR): We will release the full codebase and REAL-Colon-Reason benchmark upon acceptance, an anonymous version: https://anonymous.4open.science/r/surgical_videoqa_dt/.
9.Figure 1 and typos (R4): We will restructure Fig.1 to clearly separate the four stages with explicit input/output annotations for each component. The “NVIDIA NVIDIA” repetition and other typos will be corrected.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper addresses reasoning-intensive surgical VideoQA using digital twin representations and reinforcement learning, and reviewers generally recognize the relevance of the task, the potential value of the REAL-Colon-Reason benchmark, and the promising empirical results.
I recommend inviting a rebuttal. The authors should mainly address the following issues: clarify the novelty over prior digital twin and RL-based digital twin reasoning methods; explain why the current benchmarks are sufficient and how the new dataset was clinically curated; provide direct or better-justified comparisons with the most relevant digital twin baselines; clarify whether the comparison with plain VLMs is fair given the proposed method’s access to specialized perception tools; report missing experimental details such as standard deviation computation, baseline adaptation protocols, and clinician involvement in validating the 2,000 Q/A pairs; and state whether the architecture, weights, code, and dataset will be released.
Overall, the work is promising, but the rebuttal should focus on novelty positioning, comparison fairness, dataset validation, expert evaluation, and reproducibility.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper addresses an important and underexplored surgical VideoQA problem and receives support for its structured digital-twin formulation and experimental results. Although there are concerns about technical novelty, comparison fairness, and presentation clarity, the positive reviews and rebuttal clarifications place the work slightly above the threshold. The final version should clearly strengthen novelty positioning and baseline discussion. Recommendation: Accept.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The rebuttal was evaluated by 2/3 reviewers. Both reviewers change their scores in diverging directions one recommending accept and the other reject. Based on my own reading of the paper my opinion is that although the approach is interesting and results are compelling there are many components in the proposed pipeline that require further investigation (e.g. the depth, segentation, object detection accuracy and how this affects performance) which are not considered. Also given the rebuttal a significant amount of the paper would need to be refocused and rewritten.
My recommendation is for the paper to be rejected
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After the rebuttal, reviewers’ recommendations remain mixed, but the contribution is promising and opens discussions.