List of Papers Browse by Subject Areas Author List
Abstract
Realistic surgical simulation plays a crucial role in training novice surgeons and in the development of autonomous agents. World models can scale such simulation environments to realistic and diverse procedures by predicting future patient states conditioned on current observations and surgical actions. However, current state-of-the-art approaches often fail to satisfy key criteria required for clinical applicability, including visual realism, physically grounded interactions, and the ability to simulate scenarios beyond the training distribution. Hence, we introduce SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples motion generation from visual realism. The symbolic component, consisting of a rule-based simulator and scene graph representations, models motion dynamics and tool-tissue interactions, while a diffusion model produces realistic visual appearance, including textures and tissue deformations. We propose an inverse pairing strategy that reconstructs real surgical videos in the simulator to obtain paired simulated and real videos, which are then used to train our video diffusion model for the reverse objective of sim-to-real translation. Our experiments show both qualitative and quantitative improvements over prior work. We demonstrate that our simulator further satisfies the key criteria, including generalisation to unseen interaction geometries, improvements in downstream phase detection, and unsupervised video style transfer. The code, data, and model weights are available at: \url{https://ssharvienkumar.github.io/SWoMo/}.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2681_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026/supp/2681_supp.zip
Link to the Code Repository
https://ssharvienkumar.github.io/SWoMo/
https://github.com/MECLabTUDA/SWoMo
Link to the Dataset(s)
https://huggingface.co/SsharvienKumar/SWoMo
BibTex
@InProceedings{SivSsh_SWoMo_MICCAI2026,
author = { Sivakumar, Ssharvien Kumar AND Johnson, Akwele AND Dhingra, Anirudh AND Frisch, Yannik AND Ghazaei, Ghazal AND Mukhopadhyay, Anirban},
title = { { SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16893},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The authors propose a surgical simulation environment (world model) which decouples the simulation (performed in a 3D rendering engine) from the video-translation (diffusion model). A key part of the method is that paired image data is generated automatically from the input video, by segmenting the video data and reconstructing the same scene and motion in the simulation environment. The translation (diffusion) models are conditioned on input images and graph representations of the scene to control Evaluation shows that novel co-occurance is handled well by the model, downstream tasks benefit from the generated data and style-transfer works well.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is very well-written and clear. Though not all components can be explained in depth in the limited space, the core components are clearly described and the authors mention they want to release code.
I especially liked the clear definition of three key criteria in the introduction. If more space is allowed in the rebuttal, I could imagine that the conclusion could more specifically re-iterate how these are addressed by the method, especially criteria II (physical plausibility) is not explicitly addressed, I think (But is probably fullfilled because the simulation is derived from real data?).
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
I don’t fully understand what the Symbolic Scene Graph adds, or why. If I understand correctly, the Graphs are created from the simulation, so they effectively encode the same information as the scene and its segmentation mask. This means that the “misalignents” and “conflicting signals” between the segmentation maps and the real images that are quoted as an issue would also exist in these graphs, and passing them to the network has no conceptual advantage? I understand why the misalignment between segmentation maps and real images causes issues during the training, but I believe the reasoning “by providing explicit structural relationships” doesn’t quite explain the solution - after all, the segmentation maps also contain this information (and, I would argue, in a more explicit manner than the graph). This may be difficult to do (because it’s a deep model), but maybe the authors can explain why they think this helps in practice? (Note: I didn’t follow reference [28], maybe this would have contained the explanation)
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- In “Video Diffusion Model with Graph-Image-Video Conditioning”, the terms “x_t” and “x_{t-1}” are used to, I think, represent the various noisy versions of an image. However, since x_{t} is aready “the image for frame t” above, I suggest to use a different parameter here. For example, “x_{t,\omega}” and “x_{t,\omega-1}”.
- I don’t think I’ve ever seen the term “world model” used when the world representation is so explicit. Most work on world models I’ve seen have an implicit/hidden/deep model of the world rather than using a simulation. While I like the approach a lot and see the benefits of an explicit representation, maybe this distinction should be clarified more. In a way, the method could be considered a simulation with a video-to-video translation component, which are usually not framed as “world models”. Note: I think it might be enough to more explicitly state that, given the current state and past of the simulation, your full method only needs an tool movement, to fully predict the next state and output video frame?
- At the end of the methods section, the description of the “second stage” is in large parts describing the ControlNet method. This is acknoledged correctly in the Introduction, but I think the citation should be repeated here for clarification, i. e. for readers who don’t know ControlNet.
- Is the batching into 16 frames also used at inference? This might hinder usage in a real-time “training” setting.
- I think Fig. 2 is not referenced in the text
- I think a common problem with world models is that they only model what they’ve seen. The OOD test is interesting, but if this is for training: What happens if the user or agent makes a mistake? Do the CATARACT datasets contain “negative” examples, issues, complications?
- “seqeunce” -> sequence
- Because the authors identify border-regions between segmented structures as critical and difficult, perhaps the comparison via “bounding boxes” is not ideal. Why not use a pixel-level segmentation?
- According to Table 1, SG2VID performs second-best in most metrics. From the introduction, I could not quite understand what the advantage of the current method is compared to SG2VID - is it mainly that the segmentation method is used directly?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I only have two issues with the paper: The graph-component may be motivated even better and the usage of the term “world-model” is (albeit probably correct) somewhat different from how it’s usually used in most state-of-the-art methods. Both issues could probably be fixed by adding a few sentence or slight rewording.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes SWoMo, a neuro-symbolic world model for cataract surgery simulation that explicitly decouples interaction dynamics from visual realism. The method combines a rule-based simulator and scene graph representation to model tool motion and tool–tissue interactions, while a diffusion model translates simulated sequences into realistic surgical videos. To bridge simulation and real data, the paper introduces an inverse pairing strategy that reconstructs real surgical videos in the simulator and uses the resulting paired data for sim-to-real video generation. The approach further demonstrates generalization to unseen interaction geometries and supports applications such as unsupervised video style transfer and downstream phase recognition improvement. Overall, the main contribution is a structured and controllable framework for interactive surgical simulation that integrates symbolic dynamics with high-fidelity video synthesis.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Well-motivated neuro-symbolic design: The paper presents a clear and technically meaningful decomposition between symbolic dynamics and neural visual synthesis, which makes the framework more controllable and interpretable than purely generative approaches. Strong practical relevance for surgical simulation: By combining a rule-based simulator with sim-to-real video generation, the method targets an important real-world problem and offers a plausible path toward more realistic and interactive surgical training environments. Comprehensive empirical validation: The paper evaluates the method from multiple angles, including visual quality, conditioning fidelity, out-of-distribution generalization, style transfer, and downstream phase recognition, which strengthens the overall experimental support.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The world model claim is somewhat overstated: The core dynamics are largely governed by a rule-based simulator rather than being learned end-to-end, which makes the method closer to a neuro-symbolic simulation pipeline than a fully learned world model. The downstream benefit is relatively limited: Although the method improves phase recognition performance, the gains are modest, especially on the larger dataset, which weakens the practical impact of the claimed utility for task model training. The evaluation scope remains narrow: The experiments are restricted to cataract surgery and a limited set of phases, with no closed-loop policy learning or interactive control evaluation, so the broader usefulness of the proposed framework is not yet fully demonstrated.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a clear and well-motivated neuro-symbolic framework that combines structured simulation with high-fidelity video synthesis, and the overall idea is both technically interesting and relevant to surgical simulation. Its strengths lie in controllability, interpretability, and fairly broad empirical evaluation across synthesis quality, conditioning fidelity, OOD generalization, and downstream utility. However, the work also has notable limitations: the “world model” formulation is somewhat overstated given the rule-based dynamics, the downstream gains are modest, and the evaluation remains limited to a narrow surgical setting without closed-loop decision-making experiments. Overall, despite these weaknesses, I think the paper offers sufficient novelty and practical value to merit acceptance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes SWoMo, a neuro-symbolic world model for cataract surgery simulation that decouples interaction dynamics from visual appearance. It uses an inverse pairing strategy to generate paired sim-real data, which can better generalize to unseen surgical scenes.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The key innovation of decomposing the motion/geometry and visual appearance is interesting, compared with previous legacy of generative modelling that lacks physical grounding.
2, The experiments and evaluation are extensive and well-executed. This work has multiple evaluation metrics, comprehensive ablation studies and downstream task (phase recognition) validation.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The utilization of rule-based simulator may be limited for physical realism in complex tissue deformation or force interactions. As indicated by PhysWorld [34], modelling deformable physics is a better option in this scenario.
2.Lack of performance comparison of other strong baseline, including physWorld [34] that models deformable objects and Surgical-CD [32] that achieve sim-to-real translation via diffusion models.
3.This work lacks novelty in algorithm. It is more of a combination of prior works (including SG2VID, PhysWorld, Surgical-CD), with application on cataract surgery. It also relies on many preprocessing modules, for example, the segmentation model, motion module, SAM2 for tool masks, etc. Such complicated framework increase the probability of error accumulation and limited robustness.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a coherent system that addresses an important gap in surgical simulation. However, the score is not high because the core novelty is incremental at the algorithm level.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We would like to sincerely thank all the reviewers for their positive comments. We greatly appreciate your recognition of SWoMo’s attempt to address the three key requirements for practically relevant surgical world models: high visual realism, physically grounded interactions, and dynamic world representations that generalize beyond the training distribution, which current state-of-the-art methods do not jointly address. We also appreciate your recognition of the motivation behind our neuro-symbolic design, which decomposes symbolic dynamics from neural visual synthesis. Furthermore, we sincerely thank the reviewers for acknowledging our effort on the extensive empirical validation. We have addressed the main comments below and will gladly incorporate all suggested revisions into the final manuscript.
(R1) Added value of symbolic scene graph (SG): As shown by the ablation in Table 1 (SWoMo-✗SG), the inclusion of SGs leads to significant improvements in both visual fidelity (up to 56.6% reduction in FVD and 43.4% reduction in FID) and conditional adherence (up to 38.5% increase in BB IoU and 39.2% increase in F1). However, we agree that the explanation for the improvements introduced by the SG is currently brief, and we will clarify this further in the final manuscript. The SG is constructed such that each node represents a connected component from the segmentation map and contains only high-level information about its component, including centroid, spatial spreading, average depth, and average optical flow within the region. As a result, the SG abstracts away local pixel-level misalignments between the segmentation and video frames, especially near tool-tissue boundaries where segmentation-only conditioning can introduce smoothing artifacts and reduce synthesis quality. Since small boundary shifts or noisy pixels have limited effect on component-level attributes such as centroid, spreading, average depth and flow, the SG extracted from the predicted segmentation can remain closer to the real scene structure and the true SG, despite local segmentation errors. By incorporating SG conditioning, the model is also not constrained to strictly follow potentially misaligned pixel-level segmentation boundaries. Instead, the SG provides higher-level structural guidance, giving the generator flexibility to decide where the visual boundary should realistically appear based on the image context, leading to sharper and more coherent tool-tissue interactions.
(R2) Framing as a true world model: We thank R1 for already helping us to highlight that our approach is still considered as a world model, but with a more explicit world representation. As R1 noted, most existing world models rely on implicit or latent representations, whereas our approach explicitly models scene dynamics, which is particularly important for physically grounded interactions and improved generalisation beyond the training distribution, where purely implicit world models will struggle. In particular, given the current simulation state together with the action input, SWoMo still predicts the next state and generates the corresponding future video frame. We will clarify this distinction more clearly in the manuscript.
(R3) Comparison to prior work: We have extensively benchmarked SWoMo against multiple baselines with publicly available implementations. The baselines mentioned by R3, such as PhysWorld and Surgical-CD, unfortunately, either do not provide publicly available code or are image-based generative models designed for unpaired datasets.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper introduces a model for cataract surgery simulation that combines a rule-based simulator and a scene-graph representation with a video diffusion model.
Reviewers consistently praise the well-motivated neuro-symbolic design, the clarity of writing, the strong practical relevance to surgical training, and the comprehensive empirical validation.
Concerns include the somewhat overstated framing as a true world model given the rule-based core, modest downstream gains, evaluation confined to a single surgical domain, missing comparisons to closely related prior work, and unclear added value of the symbolic scene graph beyond segmentation maps.
Reviews are consistent in supporting acceptance.