List of Papers Browse by Subject Areas Author List
Abstract
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4727_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026/supp/4727_supp.zip
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{TriFel_ORAction_MICCAI2026,
author = { Tristram, Felix AND Özsoy, Ege AND Benz, Christian AND Walch, Marcel AND Ghazaei, Ghazal AND Navab, Nassir},
title = { { OR-Action: Multi-Role Video Understanding with Fine-Grained Actions } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16892},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper introduces OR-Action, a fine-grained, multi-role temporal action benchmark for operating room (OR) understanding, constructed by mapping scene graph annotations from EgoExOR into dense action segments using rule-based heuristics. In addition, the authors propose a vision-only multi-role temporal model based on a video foundation encoder and a multi-view-to-single-view feature alignment strategy. Experiments show that vision-based temporal modelling outperforms scene-graph-based pipelines on this benchmark.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
External OR understanding is highly relevant and challenging due to occlusions, multi-actor interactions, and temporal complexity. OR-Action provides a new evaluation setting for fine-grained, multi-role temporal actions, which is valuable for the community. The paper highlights a significant gap between scene graph prediction and temporal action understanding. The vision-only model is simple, scalable, and benefits from strong pretrained video representations. The teacher–student alignment is relevant for real-world deployment where multi-view data is limited.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Evaluation unfairly disadvantages scene-graph approaches: The benchmark assesses scene-graph models using predicted graphs fed through a rule-mapping pipeline, which can lead to error propagation and misrepresent the limitations of the scene-graph paradigm. Comparisons with graph-based baselines are imbalanced: the vision model benefits from strong pretraining, whereas the graph-temporal baselines are simpler. The omission of stronger or hybrid alternatives weakens any comparative claims. Short temporal windows limit reasoning: Modelling only ≈16s fails to capture long-term procedural dependencies, restricting the ability to analyse complex OR actions. Errors and generalisation remain unexplored: The paper lacks per-class breakdown, detailed failure analysis, and robustness checks across roles or viewpoints.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper introduces an important benchmark but lacks methodological novelty. Heuristic label generation and imbalanced evaluation weaken claims against scene-graph approaches. Conclusions require caution; the benchmark contribution adn reported result justifies weak acceptance despite these flaws.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper presents a new benchmark for ego-centric activity recognition in the OR by extending the public EgoExOR with 78 action classes. Additionally, a vision-only method is provided that outperforms graph-scene-based methods.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The work provides a relevant benchmark that involves an extensive list of to-be-detected actions in a surgical scene. 2.It is shown convincingly that the sole use of exocentric camera views for detailed surgical action detection is very challenging, while egocentric views could improve the detection results. 3.The experiments show that good performance can be retained with a single egocentric view, using alignment by a student-teacher learning framework, and selecting the right clinical role (surgeon or assistant).
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.An exhaustive list of action labels is not presented, which makes it more difficult to interpret the clinical relevance once these actions could be detected. 2.Instead of providing one qualitative example (Fig.3), it would be good to provide multiple examples with best and worst performances, to understand the full range of the method’s abilities.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a relevant new benchmark driving the field of surgical scene understanding. The presented method is rather simple, but effective, and differs from the “standard” scene graph approach. Ablation studies provide relevant insights and give direction to future work.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
They introduce vision-only temporal model (VJEPA2-based) that is better than graph-based methods when using all egocentric views, plus a multi-view teacher → single-view student feature alignment strategy to improve single-view performance and reduce the need for multi-person egocentric capture. They also mention about releasing dataset, which is usually a great contribution in future research.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Authors introduce multi-role OR benchmark for temporal action understanding enabling fine grained OR understanding. They demonstrate vision-only temporal model (VJEPA2-based) that significantly outperforms graph-based principled evaluation and a demo video. Paper is well written with apt set of experiments.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Annotation generation is heuristic and may encode “rule bias”, it is still not human verified. We can have some human in loop evaluation, like annotate random frames and evaluate. Multi-view OR / surgical action modeling exists in, e.g. Schmidt et al., 2021; clarify what is uniquely enabled by their alignment objective vs prior multi-view attention or distillation approaches.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper solves a known problem using known solution but application is novel. Authors also provide a benchmark which is a great initiative for enabling further research.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #4
- Please describe the contribution of the paper
The paper proposes OR-Action, a fine-grained, multi-role action understanding benchmark for external OR videos. The benchmark is constructed by mapping ground-truth scene graph annotations from a publicly available ego-exocentric OR dataset into dense role-specific action segments using handcrafted temporal rules. The paper further evaluates scene-graph-based approaches for this new action-recognition task and proposes a vision-only temporal model based on a frozen video foundation model, together with a multi-to-single-view feature alignment strategy to improve single-view egocentric action prediction. Results suggest that predicted scene graphs struggle to support temporally coherent fine-grained action recognition, while the proposed vision-only model achieves stronger performance.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses an important and relatively underexplored problem: fine-grained OR activity understanding involving multiple participants, tools, and devices. As the authors mention, this topic has clear relevance for workflow-aware assistance, automated documentation, surgical training, and safety monitoring. 2.The proposed OR-Action benchmark is a useful attempt to move external OR understanding beyond frame-level scene graph prediction toward role-specific action recognition. 3.The paper provides an interesting evaluation of whether scene graph representations can support temporal action understanding. The comparison involving predicted scene graphs, ground-truth scene graphs, GNN-based baselines, and the proposed vision-only model helps reveal the gap between relational parsing and robust temporal action recognition. 4.The vision-only model and multi-to-single-view feature alignment are well motivated. The method is aligned with the challenges of OR videos, including occlusion, multi-person interaction, and severe active/idle imbalance. 5.Using a multi-view teacher to improve a single-view student is a reasonable direction for reducing sensing requirements.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.(Major) The construction of the benchmark relies heavily on handcrafted rule-based mapping from ground-truth scene graphs to action labels. As a result, the derived labels may inherit the assumptions and biases of the manually designed rules. The task is also partly framed as learning to reproduce labels produced by a rule system, rather than predicting independently annotated human action labels. 2.(Major) Following the previous point, GT scene graph + rule mapping effectively defines the new ground truth for action understanding and should reproduce the labels by construction. However, GT + GNN shows significantly worse performance, which suggests that the implementation of the GNN baseline may be suboptimal (for example, due to only connecting to [t−1; t+ 1]). Key GNN implementation details, such as architecture, edge encoding, temporal connections, readout, and training protocol, are also missing. 3.The validity of the generated action labels is not sufficiently established. The paper states that generated annotations are visually verified, but it does not provide enough detail about the verification protocol or whether clinical experts were involved. 4.The proposed multi-to-single-view feature alignment appears to be a relatively straightforward feature-level teacher-student distillation objective using an L1 loss between normalized teacher and student tokens. The paper does not sufficiently position this design relative to existing cross-view contrastive learning, knowledge distillation, and other relevant alignment methods. 5.The gains from feature alignment may not necessarily show that the student learns information from other viewpoints. The improvement could also come from a general regularization or distillation effect. The authors should add control experiments, such as shuffled teacher features, random teacher features, same-view teacher distillation, or standard regularization baselines, to better support this claim. 6.The loss formulation for idle frames is unclear. The authors should clarify whether the K action classes include idle, what action label is used for the action part of the loss function, or whether the action loss is masked out for idle frames. Currently, the mask only applies to the sum of the two cross-entropy losses. 7.The paper sets the number of Role Pooler queries Q equal to the number of frames T and treats the outputs as per-frame embeddings. However, learnable query tokens do not automatically guarantee that the t-th output corresponds to the t-th video frame. The authors should clarify how this temporal alignment is ensured. 8.The paper lacks comparisons with existing vision-based action recognition methods, such as I3D, SlowFast, Video Swin, and LSTR. Since the proposed model is the only vision-based action recognition method evaluated, comparisons with established vision-based baselines would be necessary. 9.The implementation details for RP, RC, and the rule-based mapping are high-level and lack sufficient details for review. The authors should release the code if the work is accepted, as promised in the manuscript, “Benchmark and code will be released upon acceptance.”
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.Figure 1 is never referred to in the text. 2.Please clarify why Q_f needs to be defined separately instead of simply using T, since the current paper does not explore the effect of different Q_f values. 3.I recommend clarifying that GT scene graph + rule mapping is the benchmark label-generation process and would trivially reproduce the labels by construction. This would help avoid confusion between label generation and model evaluation. 4.Some notation in Section 3.1 is unclear. In particular, R_in is confusing because it seems to refer to input streams when it is defined, but later appears to refer to OR roles. The authors should better define this notation. 5.The experimental results lack statistical analysis or uncertainty estimates. Given the relatively limited number of action segments and the likely variability in training video models, reporting mean and standard deviation over multiple runs would make the conclusions more reliable. 6.Sensitivity analysis for key rule-based mapping choices is missing, such as the “recent” temporal window, event extension duration, smoothing threshold, and rule priority order. This would help assess how stable the benchmark labels are under reasonable changes to the mapping rules. 7.Statistical testing to validate the effect of the alignment is not reported.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The action labels are derived from handcrafted rule mapping rather than independently annotated human action labels. The GNN baseline and feature alignment claims are also under-validated, with missing implementation details, limited baseline comparisons. I do not believe these issues could be easily addressed during the rebuttal and camera-ready revision period.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely thank the reviewers and AC for their constructive feedback and for recognizing the relevance of OR-Action as a fine-grained multi-role benchmark for external OR understanding, as well as the practical value of egocentric video and multi-to-single-view alignment.
Regarding the heuristic annotation pipeline and potential rule bias (R1/R3/R4), we agree this is an important limitation. Our goal is not to claim that rule-generated labels are equivalent to independently annotated human action labels. Rather, OR-Action is distilled from existing expert scene-graph annotations to enable the first temporal, role-specific evaluation on this public ego-exocentric OR dataset. We tried to make the actions as hard as possible, focusing on procedural aspects that are hard to distinguish without longer temporal context to provide an interesting avenue for future research. Our results for both vision only and even oracle GT+GNN models highlight that current models are not yet equipped to fully handle multi-view multi-actor video understanding, with relatively low Edit and F1 scores compared to easier datasets.
Regarding fairness to scene-graph methods and the GNN baseline (R1/R4), we will clarify that the predicted-scene-graph results should not be read as a rejection of scene graphs as a paradigm. Our GT+GNN upper bound shows that ground-truth scene graphs are informative for action understanding; the main gap is between predicted frame-wise graphs and temporally coherent dense actions. We will try to include the GNN implementation and training details in the official benchmark release.
Regarding the vision model, temporal context, and baseline comparisons (R1/R4), our intent is to evaluate whether strong pretrained video representations with explicit temporal pooling can form an effective alternative to predicted scene graphs in this benchmark. The 64-frame / 16s window was chosen to match the video encoder setting and keep the method deployable in streaming OR scenarios. We agree that longer context may capture procedural dependencies beyond local actions, but want to clarify that running video encoders such as VJEPA2 on clips much longer than 16 seconds becomes computationally infeasible due to quadratic self-attention scaling.
Regarding multi-to-single-view alignment (R1/R3/R4), we agree that the presentation should better separate what is demonstrated from what remains a hypothesis. The results show that a single-view student can improve when aligned to a multi-view teacher, especially for clinically active viewpoints such as surgeon and assistant. We will clarify claims that the student “sees” unobserved viewpoints and instead describe the objective as distilling multi-view contextual structure when cues are present in the observed stream.
Regarding technical clarity (R4), we will clarify that idle is handled through the separate activity head, while the fine-grained action head is evaluated on active frames after masking. We will also clarify how temporal tokens are obtained by setting the Role Pooler query count to the number of sampled frames and supervising the sequence frame-wise, simplify notation such as R_in, fix the missing Fig. 1 reference.
We reaffirm that the code and annotations will be made public upon acceptance.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper introduces a fine-grained multi-role action-centric benchmark for operating room video understanding derived from an existing dataset, together with a vision-only temporal model and a cross-view feature alignment strategy.
Reviewers highlight the relevance of a fine-grained benchmark, the convincing demonstration of the vision-only temporal models, and the practical utility of the alignment scheme.
Reviewers raise concerns about potential rule-bias in the heuristic annotation pipeline, limited graph-based baselines, short temporal context windows, missing per-class and failure analyses, etc.
Reviews are consistent in supporting acceptance.