Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Surgical scene understanding is essential for AI-assisted intervention, yet real-world clinical applications require a holistic understanding of procedural context, semantic reasoning, and precise visual grounding. However, existing approaches typically address these components in isolation, leading to fragmented representations and limited semantic consistency. To address this limitation, we propose SurgMLLM, a unified surgical scene understanding framework that bridges high-level reasoning and low-level visual grounding. Given surgical videos, SurgMLLM fine-tunes a multi-modal large language model (MLLM) to support structured interpretability reasoning over phases, instrument-verb-target (IVT) triplets, and triplet-entity segmentation tokens. These tokens are then temporally aggregated and serve as prompts for a segmentation network, enabling accurate pixel-wise grounding of instruments and targets. The framework is trained end-to-end with a unified objective that couples language-based reasoning supervision with visual grounding losses. To facilitate unified evaluation, we introduce CholecT45-Scene, extending CholecT45 dataset with 64,299 frames of pixel-level mask annotations for instruments and targets, aligned with existing triplet labels. Extensive experiments show that SurgMLLM improves the primary triplet recognition metric AP_IVT from 40.7% to 46.0% and consistently outperforms prior methods in phase recognition and segmentation. These results highlight the effectiveness of unified reasoning-and-grounding for reliable, context-aware surgical assistance. Our code and dataset are available at https://github.com/Su-Llo/SurgMLLM.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2568_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: https://papers.miccai.org/miccai-2026/supp/2568_supp.zip

Link to the Code Repository

https://github.com/Su-Llo/SurgMLLM

Link to the Dataset(s)

N/A

BibTex

@InProceedings{HuaJin_Towards_MICCAI2026,
        author = { Huang, Jincai AND Zou, Shihao AND Guo, Yuchen AND Li, Jingjing AND Ji, Wei AND Wang, Kai AND Wang, Shanshan AND Si, Weixin},
        title = { { Towards Unified Surgical Scene Understanding: Bridging Reasoning and Grounding via MLLMs } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes SurgMLLM, a unified framework that bridges high-level surgical reasoning (workflow phases, IVT triplets) and low-level pixel grounding via a multi-modal LLM. It introduces entity-specific [SEG] tokens with temporal residual fusion to prompt SAM2, and releases CholecT45-Scene, a densely annotated dataset (64,299 frames) enabling joint evaluation of reasoning and grounding.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Novel Unified Architecture: First to jointly model phase recognition, triplet reasoning, and entity grounding in a single MLLM-driven pipeline, addressing the fragmentation of prior surgical scene understanding methods. Well-Motivated Design: The [SEG] token bridge and temporal fusion explicitly tackle surgical video consistency and cross-task semantic alignment. Strong Empirical Results: Consistently achieves SOTA across all three tasks on CholecT45 (e. g. , APIVT +5.3%, mIoU 84.4%), with clear ablation validating joint optimization benefits. Valuable Community Resource: CholecT45-Scene provides dense pixel-level masks and structured reasoning narratives, filling a critical gap for unified surgical video analysis.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Baseline Fairness & Currency: Comparisons mainly include methods from 2021–2024 and omit recent reasoning-segmentation frameworks (e. g. , LISA, Sa2VA, VideoLISA, MEJO) that share the [SEG]-to-SAM paradigm. Performance gains may partially stem from backbone capacity (4B MLLM + SAM2-H) rather than architectural novelty. Clinical Deployment Feasibility Unaddressed: No inference latency, FPS, or memory analysis is provided. Given the computational cost of a 4B MLLM + SAM2-H pipeline, real-time intraoperative applicability remains unclear. Annotation QA Transparency: While manual auditing is mentioned, quantitative inter-annotator agreement, SAM2 correction rates, and handling of ambiguous target boundaries (e. g. , deformed gallbladder, omentum) are not reported. Mechanistic Validation of Mutual Promotion: Table 4 confirms joint training helps, but lacks gradient/attribution analysis or error-case studies demonstrating how grounding concretely corrects MLLM reasoning errors (or vice versa).

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    I encourage the authors to address the following in the rebuttal: Add zero-shot or fine-tuned comparisons with recent grounded MLLMs (e. g. , Sa2VA, LISA) to isolate the contribution of the surgical-specific design vs. backbone scaling. Report inference latency/FPS and discuss practical deployment trade-offs. Provide quantitative annotation QA metrics (e. g. , inter-expert Dice on a held-out subset) and clarify target mask handling protocols.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper presents a timely and well-executed unified framework for surgical scene understanding. The methodological design is sound, the empirical results are strong across multiple tasks, and the released dataset is a valuable community contribution. However, the evaluation lacks direct comparison with contemporary reasoning-segmentation paradigms, and clinical deployment feasibility (latency/efficiency) is unaddressed. These gaps prevent a stronger recommendation, but the technical rigor, clear contribution, and dataset value justify a Weak Accept. Acceptance should be contingent on the authors clarifying baseline fairness, reporting efficiency metrics, and providing additional annotation QA details in the rebuttal.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper proposes SurgMLLM, a framework that couples an MLLM (InternVL2.5-4B, LoRA-adapted) with SAM2 to jointly produce (i) workflow phase predictions, (ii) surgical IVT triplet predictions, and (iii) pixel-level segmentation masks for both instruments and targets. The bridge between reasoning and grounding is a LISA-style [SEG] token: per-frame [SEG] tokens emitted after each instrument/target mention in the LLM’s structured output are projected into SAM2’s prompt space and temporally fused via residual aggregation before mask decoding. Training combines language-modeling loss on a structured / output with BCE/Dice grounding losses and an entity-aware token-weighting loss. The paper also introduces CholecT45-Scene, a dataset extending CholecT45 with 64,299 frames of pixel-level masks for both instruments and targets (previously only instruments were available in CholecTriplet-Seg), plus structured reasoning narratives generated by Qwen3 conditioned on GT labels and masks. Evaluation covers phase recognition, triplet recognition, and triplet-entity segmentation on 5-fold CV, with SOTA claimed across all three.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Dataset contribution. Extending CholecT45 with target masks fills a real gap — CholecTriplet-Seg provides instrument masks only, and target grounding for anatomical structures is exactly the piece needed to evaluate IVT grounding end-to-end. The reasoning narratives, while synthetic, enable a new supervision signal for the community.
    • Task formulation. Unifying phase + triplet + pixel-level triplet-entity grounding in a single model is, to my knowledge, new at the pixel level. The ablations in Table 4 (w/o triplet, w/o phase, w/o grounding) provide meaningful evidence that the three objectives interact constructively rather than degrade one another.
    • AP_IVT improvement. The +5.3 point gain on the primary triplet metric (40.7 → 46.0) over CurConMix is a meaningful step. Gains on AP_T (+10.9) and AP_IV (+22.2) suggest the grounding supervision genuinely helps IVT reasoning, which is a non-obvious and positive finding.
    • Grounding numbers. 84.4 mIoU vs 65.8 for SurgSAM2 is a large margin, and the ablations indicate the per-frame [SEG] + residual fusion design contributes to this.
    • Evaluation protocol. 5-fold CV with mean±std follows the established protocol, making the numbers comparable to prior work.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    W1: Overstated novelty of the “unified MLLM for surgical scene understanding” claim. The introduction and contribution list position this as the first unified MLLM framework, but this ignores GP-VLS (Schmidgall et al., 2024), which unifies phase, triplet, and tool recognition in a surgical VLM, and SurgVLM (2025) and EndoChat (MedIA 2026), which additionally incorporate bounding-box-level visual grounding alongside surgical reasoning. The contribution is more accurately described as the first MLLM framework to bridge structured triplet reasoning with pixel-level grounding of both instruments and targets via [SEG]-driven SAM2 decoding. The related work section and triplet-recognition baselines should be updated accordingly. At minimum, one surgical MLLM should appear in Table 2 — currently all triplet baselines are specialist classifiers, which makes the MLLM contribution look stronger than a head-to-head with a surgical VLM would. W2: Architectural novelty is limited and under-explained. The reasoning-to-grounding bridge reproduces the Sa2VA [26] architecture, which itself adapts LISA [13] to SAM2: expanding the LLM vocabulary with a [SEG] token, extracting its last-layer hidden state, projecting it through an MLP into SAM2’s sparse prompt embedding space, and training end-to-end with BCE + Dice + language modeling losses while fine-tuning only the SAM2 mask decoder. SurgMLLM adopts this pipeline, including the same InternVL2.5 backbone used in Sa2VA. The mechanistic contributions beyond Sa2VA appear to be the per-entity residual fusion over per-frame [SEG] tokens (Eq. 2) and the structured / reasoning format, but this inheritance is not acknowledged in either the related work or the method section. Beyond the novelty framing, the method section leaves several implementation details unspecified that are necessary to reproduce or correctly interpret the approach: (a) Which LLM representation is extracted at [SEG] positions (last hidden layer, pooled, or other)? (b) Is [SEG] a newly added vocabulary token, or a reused existing token? (c) Which SAM2 prompt slot (sparse prompt embeddings, dense embeddings, other) does the projected embedding replace or augment? (d) What does "entity u" in Eq. 2 actually group — the entity class across all frames (e.g., all occurrences of "grasper"), per triplet-slot position, or per instance? Each choice yields different fusion behavior and different handling of multiple simultaneous instances of the same class. (e) Do gradients from the mask losses flow back through the projection MLP into the LoRA adapters, or is the LLM trained purely from the language modeling and entity-aware losses? A clearer equation chain from LLM hidden state → projection → SAM2 prompt slot → mask, together with explicit positioning relative to Sa2VA and LISA, would substantially strengthen the paper. W3: The central claim that reasoning helps grounding is not ablated. The paper sells the ... block as enabling "explicit structured reasoning," but Table 4 does not contain an "answer-only" row. Without an ablation removing the CoT block while retaining phase + triplet + grounding supervision, readers cannot tell whether the reasoning narrative actually contributes to grounding quality or whether it is a stylistic wrapper on the GT labels that the Qwen3-generated supervision trained the model to reproduce. W4: mIoU in Table 3 does not reconcile with the reported IoUs. With IoU_I = 87.8 and IoU_T = 67.8, a simple mean is 77.8, not 84.4.Per-class averaging across the 6 instrument and 15 target categories would produce a different (likely pixel-frequency-weighted) aggregate. Since mIoU is the headline grounding metric, the exact definition must be stated in the manuscript. W5: Segmentation baselines are underspecified. Table 3 compares against SAM2-ZeroShot, SAM2-Adapter, and SurgSAM2, but what prompts are the baselines given? Points? Boxes? Text? If SurgMLLM receives LLM-derived entity-specific prompts while baselines receive weaker prompts, the comparison is not isolating the grounding method. W6: Multi-task vs. single-task baseline fairness. SurgMLLM benefits from phase + triplet + mask supervision simultaneously. Phase and triplet baselines receive only their own labels. The architectural contribution is not cleanly separated from the multi-task-supervision effect. W7: Circular GT for segmentation evaluation. CholecT45-Scene masks are generated by SAM2 + ~30 point prompts + temporal propagation + manual audit. SurgMLLM also uses SAM2.Systematic SAM2 failure modes are therefore baked into both the GT and the predictions, biasing IoU upward for SAM2-based methods. An externally annotated held-out subset, or cross-evaluation on CholecTriplet-Seg, would strengthen the claim. W8: AP computation from autoregressive text is not explained. Triplet AP requires per-class confidence scores. How are these derived from the LLM's text output — token logits at the entity position? Sampled probabilities? Rank-based? This is non-trivial and fundamentally affects comparability with AP-trained classifiers (RDV, TERL, CurConMix). W9: Cascaded recall is not analyzed. If the MLLM under-predicts N triplets, the missing entities are never segmented, and there is no way to separate reasoning errors from grounding errors. The grounding results would be more informative decomposed into (triplet recall) × (mask IoU \| triplet recalled). W10: No latency or compute reporting. The introduction motivates "context-aware computer-assisted intervention," implying near-real-time use. At minimum, GPU used and effective fps achieved should be stated. W11: Temporal consistency is claimed but not measured. Residual fusion is motivated as a temporal-consistency mechanism, but only frame-wise mIoU is reported. W12: L_ent is introduced as a contribution but never ablated, loss weights (λ_bce, λ_dice, λ_ent) are not reported, and the temporal fusion uses an unspecified number of frames T. The "lightweight MLP aggregation" is not described architecturally. W13: Reasoning narrative generation is under-described. The main text does not specify, the paper only references "Qwen3" in a figure caption. The specific model version, the prompt template, and any validation of factual correctness of generated narratives are missing. Given that the narratives are a named supervision signal, this is a reproducibility gap. W14: Dataset construction protocol is thin. Number of annotators, inter-annotator agreement, audit sampling rate, rejection criteria, and the fraction of propagated masks that were corrected are not reported. The 90,489 → 64,299 frame reduction from CholecT45 to CholecT45-Scene is also unexplained and may introduce selection bias. W15: Ablations are single-fold (Fold 1) while main results are 5-fold. Single-fold ablations are noisy. The suspiciously low std on Phase Accuracy (0.6 vs 2.3–4.2 for baselines) is also worth explaining. W16: Results narrative is one-sided. The text claims consistent improvements across Table 2, but AP_I drops from 93.5 (TERL) to 89.9 and AP_V drops from 72.8 (TERL) to 71.0.These regressions deserve honest discussion. (AP_IVT is the primary metric and the +5.3 gain there is substantial, but the paper should discuss the component losses.)

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    The paper tackles a well-motivated problem and proposes a sensible solution. The task formulation, unifying phase recognition, IVT triplet reasoning, and pixel-level triplet-entity grounding in a single framework, is a natural next step for surgical scene understanding, and the AP_IVT improvement (+5.3 over CurConMix) is a meaningful result. CholecT45-Scene fills a real gap by adding target masks alongside instrument masks and aligned reasoning narratives, and it will likely be useful to the community. The ablations in Table 4 support the claim that the three objectives interact constructively. Overall, the method makes sense, the motivation is solid, and the results are strong. My reservations concern the presentation and rigor of several specific claims rather than the substance of the work. Each of the weaknesses listed in W1–W16 is individually addressable, but their cumulative weight is what places the paper below the acceptance threshold in its current form. I would particularly ask the authors to resolve the following in the rebuttal and camera-ready:

    W1 (novelty framing). Please update the related work and contribution framing to reflect prior surgical MLLM work (GP-VLS, SurgVLM, EndoChat). The contribution is more accurately the first MLLM framework to bridge structured triplet reasoning with pixel-level grounding of both instruments and targets via [SEG]-token-driven SAM2 decoding. Ideally, at least one surgical MLLM should appear as a baseline in Table 2.W2 (bridge implementation details). A cleaner method subsection specifying which LLM representation is extracted at [SEG], whether [SEG] is an added vocabulary token, which SAM2 prompt slot it occupies, how Eq. 2 groups entities across frames, and how gradients flow between the modules would significantly improve clarity and reproducibility. W3 (no CoT ablation). The block is sold as the central reasoning mechanism, yet it is never ablated. An "answer-only" row in Table 4 is needed to substantiate the claim that structured reasoning narratives contribute beyond the label supervision. W4 (mIoU definition). The reported 84.4 mIoU does not reconcile with IoU_I = 87.8 and IoU_T = 67.8.Please state the aggregation rule explicitly. W8 (AP from autoregressive text). Please specify how per-class triplet confidence scores are derived from the LLM's text output for AP computation. Without this, Table 2 is not directly comparable to AP-trained baselines (RDV, TERL, CurConMix).

    The remaining weaknesses (W5–W7, W9–W16) should be addressed appropriately in the rebuttal or camera-ready, though I consider them less critical individually. In particular: specifying prompt types for the SAM2 baselines in Table 3 (W5), reporting GPU and effective fps (W10), clarifying the Qwen3 variant and narrative-generation prompt (W13), clarifying the 90,489 → 64,299 frame reduction (W14), and discussing the AP_I and AP_V regressions honestly in the text (W16) would all strengthen the paper. Minor: please specify the number of input frames T used at training and inference, the loss weights (λ_bce, λ_dice, λ_ent), and the temporal fusion architecture. I want to emphasize that the underlying work is strong. If the rebuttal convincingly resolves W1–W4 and W8 and addresses the remaining points appropriately, I would be comfortable moving to Accept.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My Weak Reject is driven by the cumulative weight of the weaknesses rather than any single fatal flaw. Each is individually addressable, but several together affect how the central claims can be interpreted.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposes a unified framework for surgical scene understanding that jointly models phase recognition, instrument-verb-target (IVT) triplet recognition, and triplet-entity grounding within a single MLLM-based architecture. The authors utilize the MLLM to generate structured surgical scene descriptions and insert entity-specific [SEG] tokens for instruments and targets, which are then projected into a SAM2-based decoder for pixel-level segmentation. In addition, the paper introduces a temporal fusion mechanism to improve consistency of grounded entities across frames. Another important contribution of the paper is the construction of Cholec T45-Scene, an extension of Cholec T45 with 64,299 frames of instrument and target mask annotations, together with structured reasoning narratives, enabling unified supervision and evaluation across phase, triplet, and grounding tasks. Comprehension experiments demonstrate the effectiveness of the proposed method.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    (1) The paper addresses an important and well-motivated problem. The unified formulation of phrase recognition, triplet recognition, and segmentation in a well-designed framework is clinically meaningful. (2) The proposed framework is well designed and conceptually coherent. The authors attempt to explicitly bridge high-level reasoning and low-level grounding. (3) The dataset contribution is valuable. The introduction of Cholec T45-Scene is a strong practical contribution to this paper. The dataset can be used to support future work on more integrated surgical video understanding. (4) The overall presentation is well organized. The problem setup, framework overview, and experimental sections are easy to read. The experimental evaluation is convincing.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    (1) The unified surgical formulation is interesting, but I think the core design ideas are related to existing grounded reasoning and segmentation frameworks, such as LISA, GLAMM, and Sa2VA. These works also connect language-side reasoning with segmentation outputs. The authors overstate the methodological novelty rather than the surgical-specific integration of phase, triplet, and grounding tasks. (2) For the grounding task, the authors mainly compare with SAM2-ZeroShot, SAM2-Adapter, and SurgSAM2, but do not include direct comparison against the more task-relevant surgical works [1, 25] mentioned in the paper. (3) The new dataset is promising, but annotation details are somewhat limited. (4) The authors should provide an anonymized GitHub repository (or other anonymous project page) containing at least the evaluation protocol and annotation data samples for verification.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I think this paper has several clear strengths, it address an important problem, proposes a coherent unified framework, introduce a useful dataset extension, and comprehensive experiments. The idea of jointly modeling phase recognition, IVT triplet reasoning, and grounded segmentation is meaningful. The experimental section is reasonably thorough, and the paper is generally well written. However, the methodological novelty relative to recent grounded MLLM work is somewhat incremental, the comparison with the most relevant prior surgical grounding methods is not fully complete.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #4

  • Please describe the contribution of the paper

    The work proposes a unified framework for surgical scene understanding for phase recognition, triplet detection, and segmentation.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • A end-to-end unified framework
    • Novel (incremental) dataset that now includes segmentation annotation for target tissue
    • In a 5-fold cross-validation test, the proposed model outperformed single-task models in most cases
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Only validated on one dataset. Therefore, there could be potential dataset bias
    • lack SOTA comparision against sub-modules, different LLM, or segmentation decoder.
    • lack of ablation studies on instrument/tissue-specific performance.
    • limited details on the reasoning (triplet/text to segmentation) and limited details on how the [seg] together are combined
    • limited SOTA comparision against surgical foundation models
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    additional ablation studies or SOTA could further clarify and strengthen the paper. However, in my view, this work has reasonable novelty and supporting experimental results for a weak accept.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We sincerely thank the reviewers and the Area Chair for their constructive feedback. We appreciate the recognition of our well-motivated problem and sensible solution (R2&R3), timely unified framework for surgical scene understanding (R1&R4), and valuable CholecT45-Scene dataset that fills a real gap (R1-R4). We first address common questions (CQs), followed by point-by-point responses.

(CQ1) Dataset annotation: Initial masks were generated by SAM2 using positive/negative point prompts, then audited by three expert surgeons with consensus correction. The audit-flag rate was 42.2%, and 32.7% of flagged frames were corrected after manual re-filtering/re-annotation. For ambiguous boundaries, prompts were used when regions were visually separable. For deformation, occlusion, or tissue overlap, annotations followed surgical semantic consistency rather than strict exclusive pixel ownership; thus, independent binary entity masks may overlap.

(CQ2) Method Novelty: Our framework extends generic LISA/Sa2VA-style reasoning segmentation to surgery by unifying three surgical tasks with frame-wise per-entity [SEG] representations, temporal residual fusion, structured Phase-IVT reasoning, and surgical-entity-aware auxiliary supervision. Unlike existing surgical MLLMs (GP-VLS, SurgVLM, and EndoChat), which mainly address VQA, our method jointly performs Phase-IVT reasoning and pixel-level grounding for both instruments and targets. Generic Sa2VA provides only global video-level responses but lacks frame-wise per-entity modeling and structured Phase-IVT reasoning. We will clarify these surgery-specific contributions in the revision.

(CQ3) Comparison with Other Baselines: On Fold 1, Sa2VA achieves 30.3 AP_IVT, compared with 46.4 AP_IVT by our method under the same backbone, demonstrating the effectiveness of our surgery-specific design. Task-relevant surgical works [1,25] do not release code, making reproducible comparison infeasible. We therefore compare against stronger SAM-series models to provide a fair and rigorous evaluation of our framework.

(R1 W1) Please see CQ2&3. (R1 W2) SurgMLLM runs at 2.8 FPS on an NVIDIA RTX PRO 6000.We will explore lighter variants to improve real-time applicability in our extension work.

(R1 W3) Please see CQ1. (R1 W4) Mask losses are not detached and back-propagate through projection, temporal fusion, and trainable LLM LoRA adapters.

(R2 W1) Please see CQ2&3. (R2 W2) Please see CQ2 and (R1 W4). In Eq. 2, entity group u denotes the same grounded phrase across frames, e.g., “grasper” or “gallbladder”, based on phrase/entity-label consistency over frame-wise [SEG] embeddings, not persistent instance ID or triplet-slot position.

(R2 W3) Removing ... while keeping the same phase, triplet, and grounding supervision yields 79.1/44.2/82.8 (Phase Acc./AP_IVT/mIoU), compared with 84.0/46.4/83.6 for the full model, showing structured reasoning effectiveness.

(R2 W4) mIoU is the average IoU over all grounded entity masks, with instrument/target ratios of 0.83/0.17; IoU_I and IoU_T report the instrument and target subsets separately.

(R2 W8) AP confidence is computed from the autoregressive generation logits. At each generated entity-token position, we apply softmax over the vocabulary logits and take the probability assigned to the generated token.

(R3 W1) Please see CQ2. (R3 W2) Please see CQ3. (R3 W3) Please see CQ1. (R3 W4) We will release the code and annotations in the camera-ready version.

(R4 W1) CholecT45 is the most representative dataset and is widely used in prior works [9–11]. We will include more datasets in the extension.

(R4 W2) Please see CQ3. (R4 W3) We report only AP_IVT due to space limit and will include other metrics in the final version.

(R4 W4) Please see CQ2. (R4 W5) Please see CQ3.



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    SurgMLLM jointly produces phase prediction, IVT triplets, and pixel-level grounding via per-frame [SEG] tokens with residual temporal fusion, and releases CholecT45-Scene (64,299 frames with dual masks and reasoning narratives). Three of four reviewers vote weak-accept on the strength of substantial empirical gains (+5.3 AP_IVT over CurConMix; 84.4 vs. 65.8 mIoU vs. SurgSAM2), the dataset value, and meaningful ablation in Table 4.novelty relative to Sa2VA/LISA/GP-VLS/SurgVLM/EndoChat should be more clearly positioned, the block contribution should be ablated, the mIoU 84.4 should be reconciled with IoU_I=87.8/IoU_T=67.8, AP from autoregressive text must be specified, segmentation baseline prompts must be documented, and single-fold ablations should be reconciled with 5-fold main results.The authors should address these points in the camera-ready version.



back to top