Abstract

In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow. Validated on both open-source and in-house datasets, our framework achieves superior simulation fidelity and physical accuracy compared to state-of-the-art methods, underscoring its potential to advance robot-assisted surgical applications.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3071_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: https://papers.miccai.org/miccai-2026/supp/3071_supp.zip

Link to the Code Repository

https://github.com/ChangjingLiu/EndoGSim

Link to the Dataset(s)

EndoNeRF: https://docs.google.com/forms/d/e/1FAIpQLSfM0ukpixJkZzlK1G3QSA7CMCoOJMFFdHm5ltCV1K6GNVb3nQ/viewform

BibTex

@InProceedings{LiuCha_EndoGSim_MICCAI2026,
        author = { Liu, Changjing AND Huang, Yiming AND Bai, Long AND Cui, Beilei AND Ren, Hongliang},
        title = { { EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a new framework for dynamic surgical scene reconstruction based on physical modelling, 4D Gaussian Splatting, and multimodal large language models, where the material field is introduced for simulation. Extensive experiments demonstrate the effectiveness of the proposed framework.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1 The proposed framework integrates the Material Point Method with 4D Gaussian Splatting and introduces the multimodal large language model to achieve great performance on the dynamic surgical scene reconstruction.

    2 The presentation of the paper is clear.

    3 The authors provided extensive qualitative results, including figures and videos, to validate the effectiveness of the proposed framework.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1 It would help to clarify the motivation for using multimodal large language models to infer initial material attributes.

    2 It would be best to use more metrics, such as standard visual quality metrics for rendered images measured as PSNR and SSIM.

    3 The analysis of experimental results is not in-depth enough. Specifically, the authors may need to explain why the proposed framework doesn’t achieve the best performance in some settings. The results of the cutting sample in Table 2 may need to be checked.

    4 It would be better to provide the ablation study on public datasets.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My recommendation is weak accept because the proposed framework is novel and meaningful and this paper provides extensive qualitative results. However, the quantitative results need further improvement, and some details need further clarification.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The manuscript proposes a method to reconstruct surgical scenes in 4D, derive material properties, and enable simulation of new scenarios on the obtained representation. It initializes material properties using a segmentation model and an MLLM, and then further optimizes material parameters through differentiable rendering and the material point method. This enables photorealistic simulations grounded in physical properties.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The choice of methodology is principled and addresses a highly relevant problem

    • The design choices are well-motivated and backed by a thorough evaluation and ablations (e.g., Tab. 4 demonstrating the non-obvious benefit of MLLM initialization)

    • Results overall are convincing and show a clear benefit (Tab. 2). The qualitative evaluation by 33 surgeons (Tab. 3) is also appreciated and matches with impressions from the instructive supplementary material

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The manuscript claims that modeling of “permanent tearing under surgical forces” is possible. How would this work with an emerging “gap” revealing underlying tissue? 4DGS can perform pruning and cloning, but it tends to simply stretch Gaussians to satisfy photometric losses rather than implicitly identifying and correctly handling emerging and disappearing objects.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The manuscript addresses a highly relevant problem with an interesting, principled, and thoroughly evaluated methodology. I recommend the paper for acceptance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper presents EndoGSim, a novel approach for achieving more physically plausible deformations in reconstructed endoscopic scenes. The proposed framework is evaluated on existing benchmark datasets as well as a newly collected in-house dataset.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The proposed framework addresses an important and relatively underexplored problem in endoscopic scene reconstruction, namely the modeling of physically plausible deformations. The approach is interesting and offers a novel contribution in this direction. The paper is generally well written and provides a decent experimental evaluation, with results that appear promising and supportive of the method’s effectiveness. Overall, the work is well motivated and thoughtfully explores a meaningful gap by introducing a more realistic way to simulate physical behavior in reconstructed endoscopic scenes.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    W1.The paper lacks sufficient information regarding computational cost. In particular, it is unclear what the computational overhead of the proposed method is compared to related approaches, and the training time for the full pipeline is not reported. W2.There is limited discussion of how the proposed physical modeling interacts with reconstruction quality. It remains unclear whether the joint use of 4DGS and MPM estimation is performed simultaneously, and whether this integration has any negative impact on the quality of the reconstructed scene compared to standard 4DGS. W3.The statistical analysis is insufficient. For Table 3, only mean surgeon-evaluated physical scores are reported, without standard deviation or other measures of variance. This makes it difficult to assess the robustness of the reported improvements. Additionally, statistical significance tests would strengthen the validity of the claimed performance gains. W4.The manuscript could better justify the manual initialization strategy for material properties. It is not fully explained how these initial values were selected, nor why this configuration leads to the weakest performance compared to the learned or optimized alternatives. W5.The paper is missing limitations and failure cases.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The proposed framework is interesting and represents a novel contribution to the problem of physically plausible deformation in endoscopic scene reconstruction. The paper is generally well written and supported by a decent experimental evaluation, which suggests promising effectiveness of the method. My decision for a weak accept is mainly driven by the novelty and relevance of the problem addressed, as well as the overall strength of the proposed idea and results. However, there are several uncertainties and missing details discussed in the weaknesses section. Addressing these points during the rebuttal could further strengthen the paper.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

Reviewer #1 We greatly appreciate your constructive comments and positive evaluation of our work. The concise responses are listed below. Response to Q1: Thank you for this valuable suggestion. Our motivation is that, by leveraging rich prior knowledge, MLLMs can predict initial material properties more accurately than manual reasoning. Such initial values can better guide the subsequent refinement and facilitate physically accurate simulation. we will clarify this in the revised paper. Response to Q2: We thank you for this insightful suggestion. This work mainly focuses on physical simulation realism, so we use RE to quantify the relative error of physical property estimation, while FID and EPE are adopted to assess the overall simulated visual performance. We plan to adopt PSNR and SSIM in our future evaluations to further complement existing quantitative results. Response to Q3: Thank you for your valuable and insightful comment. We will supplement the explanations for cases that do not attain the best performance. Response to Q4: Thank you for your valuable suggestion. We will conduct relevant ablation studies on public datasets in our future work.

Reviewer #2 We sincerely thank you for your careful review and positive feedback on our work. Below are our detailed responses to your concerns. Response to Q1: Thank you for your thoughtful and insightful question. In our framework, permanent tearing mainly occurs in physics simulation. Specifically: (1) Permanent tearing can be realized at the physical simulation level using the Material Point Method (MPM). When surgical forces induce plastic deformation, MPM material points naturally separate following continuum mechanics, forming a “gap” at the physical level. (2) Since each Gaussian is tightly bound to an MPM material point and its attributes are driven by the physical state via P2G/G2P transfers, the Gaussians separate along with the material points, naturally forming a visible “gap”. At the visual rendering level, the newly exposed internal tissue may exhibit some distortion or artifacts. Thus, building upon the current physics-based tearing, we plan to further improve the visual rendering consistency in future work.

Reviewer #4 We sincerely thank the reviewer for the constructive feedback. We greatly appreciate the reviewer’s recognition of our work, and the insightful suggestions are highly valuable for further improving the manuscript. The following are our responses to your questions. Response to Q1: We thank you for your valuable suggestion. We will add more detailed descriptions of the computational cost in the revised manuscript. Response to Q2: Thank you for the valuable suggestion. Our framework first completes scene reconstruction and then performs MPM physical estimation sequentially for simulation. Therefore, reconstruction accuracy and the precision of physical parameter estimation both affect the final simulation performance. Response to Q3: We greatly appreciate your insightful comment. We will include more evaluations on statistical analysis in the revised manuscript. Response to Q4: Thank you for your comment. Manual initial material values are empirically defined based on general physical priors. Improperly set manual parameters may induce numerical instability in the simulation, which consequently yields inferior performance. We will add relevant explanations in the revised manuscript. Response to Q5: Thank you for your valuable suggestion. We will supplement more discussions on the limitations and typical failure cases in the revised manuscript.

# Meta-Reviewer #2 We sincerely thank the meta-reviewer for the thorough evaluation and positive recommendation. We greatly appreciate your constructive feedback and insightful comments, and we will carefully address your valuable suggestions in the revised manuscript.




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The reviewers unanimously recommend the acceptance of this work, highlighting the paper’s significant technical novelty in integrating 4D Gaussian Splatting with the Material Point Method and MLLMs for physically grounded surgical reconstruction. The framework is highly relevant to the CAI community, well-presented, and supported by a compelling user study involving 33 surgeons. While the qualitative results are strong, final acceptance is contingent on the authors revising the manuscript to include standard visual quality metrics (PSNR/SSIM), reporting statistical significance for the user study, and clarifying the computational overhead of the pipeline. Additionally, the authors should provide a more thorough discussion of the system’s limitations, particularly regarding its ability to handle topological changes like tissue tearing and its performance fluctuations in specific experimental settings.



back to top