List of Papers Browse by Subject Areas Author List
Abstract
Free-text promptable 3D medical image segmentation offers an intuitive and clinically flexible interaction paradigm. However, current methods are highly sensitive to linguistic variability: minor changes in phrasing can cause substantial performance degradation despite identical clinical intent. Existing approaches attempt to improve robustness through stronger vision-language fusion or larger vocabularies, yet they lack mechanisms to consistently align ambiguous free-form expressions with anatomically grounded representations. We propose Skill-Evolving groundEd Reasoning (SEER), a novel framework for free-text promptable 3D medical image segmentation that explicitly bridges linguistic variability and anatomical precision through a reasoning-driven design. First, we curate the SEER-Trace dataset, which pairs raw clinical requests with image-grounded, skill-tagged reasoning traces, establishing a reproducible benchmark. Second, SEER constructs an evidence-aligned target representation via a vision-language reasoning chain that verifies clinical intent against image-derived anatomical evidence, thereby enforcing semantic consistency before voxel-level decoding. Third, we introduce SEER-Loop, a dynamic skill-evolving strategy that distills high-reward reasoning trajectories into reusable skill artifacts and progressively integrates them into subsequent inference, enabling structured self-refinement and improved robustness to diverse linguistic expressions. Extensive experiments demonstrate superior performance of SEER over state-of-the-art baselines. Under linguistic perturbations, SEER reduces performance variance by 81.94% and improves worst-case Dice by 18.60%. Project resources are available at: https://seer-medseg.github.io
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3761_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://seer-medseg.github.io/
Link to the Dataset(s)
https://seer-medseg.github.io/
BibTex
@InProceedings{ZhaTon_SkillEvolving_MICCAI2026,
author = { Zhang, Tongrui AND Wang, Chenhui AND Li, Yongming AND Chen, Zhihao AND Zhan, Xufeng AND Shan, Hongming},
title = { { Skill-Evolving Grounded Reasoning for Free-Text Promptable 3D Medical Image Segmentation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16884},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper introduces Skill-Evolving groundEd Reasoning (SEER), a framework designed to solve the “linguistic brittleness” of free-text promptable 3D medical image segmentation. The authors identify that current models are highly sensitive to minor phrasing changes (synonyms, abbreviations, typos) even when clinical intent is identical. The primary technical contributions include:SEER-Trace Dataset: A curated corpus of 22,330 multimodal instruction instances from 1,811 cases, providing “process supervision” through image-grounded, skill-tagged reasoning traces. Grounded Reasoning Chain: A vision-language inference module that extracts anatomical evidence ($e$) and formulates a rationale ($r$) before generating a canonical executable specification ($a$) for the segmentation backbone. SEER-Loop: A dynamic self-refinement mechanism using Group Relative Policy Optimization (GRPO) to distill high-reward reasoning trajectories into a reusable “SEER-Bank” of skills. Empirical Results: Significant performance gains, including an 81.94% reduction in performance variance under linguistic perturbations and high zero-shot performance on the out-of-distribution (OOD) PENGWIN dataset.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Conceptual Novelty (Active Reasoning): Unlike prior “black-box” fusion models like VoxTell or MedSAM3 that map text directly to features, SEER enforces an auditable reasoning step that verifies clinical intent against patient-specific anatomical evidence.
Novel Reinforcement Learning Application: The adoption of Group Relative Policy Optimization (GRPO) is highly sophisticated. By eliminating the need for a separate critic model and optimizing for both accuracy and stability (Equation 3), the authors effectively address the core issue of prompt sensitivity.
Dataset Contribution: The SEER-Trace dataset fills a critical gap in medical AI by providing “thinking traces” rather than just image-mask pairs. This supports higher-order supervision for anatomical localization and spatial reasoning.
Exceptional Robustness: The results on the PENGWIN dataset (Strictly OOD) are particularly strong, achieving a mean Dice of 97.39 with a standard deviation of only 0.98 under perturbations. This demonstrates that the reasoning module effectively “immunizes” the segmenter against wording changes.
Cross-Backbone Generalization: The ability to use SEER as a “front-end” for a frozen MedSAM3 model demonstrates impressive modularity and utility for the broader MICCAI community.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
3D Spatial Continuity Concerns: The reasoning phase relies on a multi-view 2D rendering $\mathcal{R}(V)$. While efficient, this projection inherently loses 3D topological information, which might be critical for segmenting complex, connected structures like vascular trees or fine-grained neuroanatomy. Teacher Bias in Dataset Curation: SEER-Trace is synthesized using a “high-capacity VLM”. There is a risk that the student model inherits idiosyncratic biases or hallucinations from this teacher. The 5% expert review sample is somewhat small for a 22k+ instance dataset. Computational Overhead: Although GRPO is more efficient than PPO, the “evolutionary rounds” in the SEER-Loop (sampling multiple trajectories, deduplication, and distillation) likely require significant compute. Taxonomy Breadth: The “skill” set is currently limited to a few categories (localization, spatial relations, synonym normalization). Real-world clinical reasoning often requires temporal comparisons or multi-modal fusion logic which are not yet addressed. Refinement Logic: The deduplication strategy in SEER-Bank (Equation 6) is not fully detailed. A lack of clarity on the semantic thresholds for merging skills could lead to either “memory bloat” or the loss of fine-grained expertise.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(6) Strong Accept — must be accepted due to excellence
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper represents a transformative step in making medical foundation models “clinically safe” by addressing linguistic instability. The methodological integration of GRPO for reasoning-driven segmentation is highly novel and timely. The quantitative evidence—specifically the 81.94% reduction in performance variance—is a definitive breakthrough that justifies a top-tier acceptance. It perfectly aligns with the MICCAI 2026 “Concept to Clinical” theme.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes SEER (Skill-Evolving Grounded Reasoning), a framework for free-text promptable 3D medical image segmentation that aims to address the sensitivity of existing text-based segmentation methods to linguistic variability. The core idea is to wrap a frozen, pre-trained segmentation backbone (primarily VoxTell) with a VLM-based reasoning module that harmonizes ambiguous free-text clinical prompts into well behaved prompts before passing them to the backbone. The system has two major parts:
- Grounded Reasoning module: Takes multi-view 2D slices from the 3D volume and the raw text prompt as input, then uses a fine-tuned VLM (Qwen3-VL) to generate structured evidence, rationale, and a cleaned-up executable answer for the downstream backbone
- SEER-bank/loop: A mechanism that stores previous successful prompt mappings in a memory bank, which can be accessed at inference time to improve future prompts
Additionally, the authors curate and claim to release the SEER-Trace dataset, consisting of 22K multimodal instruction instances pairing raw clinical requests with skill-tagged reasoning traces.
Important note: The contribution should be more explicitly framed as a prompt refinement/augmentation system built on top of existing frozen segmentation backbones, not as a standalone segmentation model, in order to properly contextualize novelty and the scope of the claims.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
Clinically relevant problem: Sensitivity to linguistic variability in free-text prompts is a real and under-addressed problem. The framing around current models suffering from “disparate embedding neighborhoods” for clinically synonymous terms is well-motivated and backed by citations
-
Clever two-stage architecture: The combination of a grounded VLM reasoning chain (forcing the model to verify anatomy visually before generating a prompt) with a dynamically evolving prompt bank is a conceptually clean and interesting design
-
Convincing ablation study (Table 2): The progressive ablation on the OOD PENGWIN dataset clearly shows that (a) naively prepending a vanilla VLM hurts performance (interesting negative result which could benefit from more explanation), (b) grounded reasoning significantly helps, and (c) SEER-Loop provides additional robustness gains. The ablation holds on the OOD benchmark
-
The mean Dice does not change too much and only a slight improvement is visible in the testing datasets. However, rather than just reporting mean Dice, the authors emphasize worst-case Dice and standard deviation under prompt variability. These present metrics that matter for the stated problem with convincing improvements. Authors can show less variance to synonym prompts and substantially higher worst-case Dice largely preventing complete failures
-
Cross-backbone generalization: The experiment replacing VoxTell with MedSAM3 as backbone, while keeping SEER’s front-end frozen, supports the claim of generalizability, although the total metric values drop significantly
-
SEER-Trace dataset: A dataset of 22K instances with structured reasoning traces and skill tags could be a useful community resource
-
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
-
Only 2 evaluation datasets is a notable limitation. VoxTell (the primary backbone used) was evaluated on more than a dozen datasets. Restricting evaluation to BrainMetShare and PENGWIN makes it hard to judge how broadly the gains generalize, especially for anatomies/modalities not covered here. Authors should at minimum discuss this scoping decision
-
Free-text prompt generation is underspecified: The prompts are described as “synthesized by an instruction-tuned large language model to mimic heterogeneous clinical contexts” however using which LLM? With what system prompt/instructions? How diverse are the resulting prompts really? This is a critical detail for reproducibility and for judging the validity of the evaluation. Have the authors considered using prompts written by actual radiologists rather than LLM-synthesized ones? The 5% expert review is noted and appreciated, but it doesn’t fully substitute for this. The prompt generation methodology needs to be reported much more carefully
-
Notation issues in the methods section: π_θ is used in Eq. (2) and (4) without ever being explicitly introduced (presumably it’s the VLM parameterized by θ?). Similarly, G in Eq. (3) is not defined in the text (presumably ground truth segmentation mask). Minor things but should be fixed.
-
Methods section is overly complex/mathy for what it describes: The paper introduces a lot of notation and operators (ξ, Ξ+, E, Dedup, B(t), ηj, etc.) to describe what is essentially a prompt refinement system with memory. This does not aid understanding, it obscures it. Novelty ≠ complexity. A clearer, more intuitive description of the system would make the contribution much more accessible
-
Eq. (7) defines a per-skill marginal utility metric ∆(s) but I couldn’t find any results or analysis in the experiments section that actually report or analyze this quantity. If you go to the trouble of formalizing it, show what you found: which skills are most useful? How does the bank evolve?
-
Limitations section is missing: The model is fundamentally bottlenecked by the frozen backbone. VoxTell has known limitations (e.g., reportedly struggles with knee structures). Can SEER compensate for inherent backbone limitations? The MedSAM3 cross-backbone experiment suggests the gains are much smaller with a weaker backbone (19.97 vs. 97.39 Dice), which is expected but also means the system’s ceiling is hard-set by the backbone. This should be discussed honestly
-
The OOD claim for PENGWIN needs qualification: The statement that PENGWIN “serves as a strictly OOD benchmark with entirely unseen anatomical regions” is only true for SEER’s reasoning module. The frozen backbone (VoxTell) may well have seen pelvic anatomy during its own pre-training. This conflation should be addressed
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper tackles a timely and clinically relevant problem, and the core idea, i.e. using visually-grounded VLM reasoning to normalize free-text clinical prompts before passing them to a frozen segmentation backbone, is sound. The robustness gains in worst-case Dice and variance reduction are compelling, and the ablation study is well-executed.
That said, several issues hold it back: evaluation is limited to only 2 datasets, the free-text prompt generation methodology is underspecified (which partially undermines evaluation validity), the methods section is unnecessarily heavy on notation, and a limitations section is missing. Most importantly, the contribution framing needs to be sharpened upfront: SEER is a prompt-augmentation wrapper around existing frozen backbones, not a new segmentation model.
Main rebuttal asks: (1) clarify the nature of the contribution explicitly, (2) detail how evaluation prompts were constructed, (3) comment on or add more evaluation datasets, (4) add/comment on a limitations section. These are largely fixable, hence weak accept pending clarification!
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The authors propose SEER, a framework for free-text promptable 3D medical image segmentation. It utilizes a Vision-Language Model (VLM, optimized with SFT and GRPO) to generate explicit
, , and reasoning traces. Furthermore, it introduces a dynamic skill-evolving strategy (SEER-Loop) that distills high-reward execution episodes into a reusable memory bank (SEER-Bank) to improve segmentation robustness against linguistic variability. - Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses a highly critical and practical clinical issue: the segmentation instability caused by linguistic variability and ambiguity in free-text prompts. 2.Decomposing clinical intent into explicit visual evidence and rationales significantly improves interpretability over traditional black-box text-to-feature conditioning. 3.Adapting advanced LLM-agent techniques (like GRPO and dynamic skill banks) to medical image segmentation is highly novel and inspiring for the community. 4.The reported improvements in worst-case Dice (Table 1) convincingly demonstrate the effectiveness of the proposed modules. Table 2 effectively proves that the performance gains stem from the proposed grounded reasoning and SEER-Loop, rather than merely the capacity of the underlying VLM (as the “+ Vanilla VLM” actually degrades performance).
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The SEER-Trace dataset relies heavily on synthetic annotations: only 5% of instances were reviewed by experts. The quality and clinical plausibility of the remaining 95% are unverified, which may introduce systematic biases or hallucinations into the training process. 2.While Section 2.3 defines the skill tuple, the actual implementation of the operators (Distillation, Dedup, Retrieval) and multi-view rendering remains conceptual. The manuscript lacks these technical specifics, hindering reproducibility. 3.The skill‑evolving mechanism has three issues. First, Eq. (7) computes marginal gain without controlling for input difficulty, causing confounding bias: skills for inherently easy cases (e.g., large organs) will appear beneficial while skills for hard, long‑tail cases may be wrongly pruned. Second, the deduplication operation and bank‑size scaling are not evaluated—no ablation or performance‑vs‑bank‑size analysis is provided. Third, the claim of “continuous adaptation in real‑world clinical settings” is overclaimed, as the reward mechanism requires ground‑truth masks, restricting it to offline training. 4.Table 1 shows OOD (PENGWIN) performance substantially higher than domain-shift (BrainMetShare) under free-text mode. This counter-intuitive result is not discussed.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.Please clarify the precise data structure of the “skills” and the implementation of the distillation operator. If an LLM is used to summarize episodes into these text artifacts, what is the exact meta-prompt used? How are these text skills concatenated with the user query during retrieval? 2.Could you comment on the inference latency? Running a VLM with iterative retrieval and reasoning before the segmentation backbone likely introduces overhead compared to end-to-end baselines.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This is a strong, forward‑looking paper that successfully brings VLM‑agent concepts to medical image segmentation. The robustness gains against linguistic variability are impressive. My main reservations, which currently keep my score at Weak Accept, revolve around several key concerns regarding the synthetic dataset, missing implementation details, design flaws in the skill‑evolving mechanism, and an undiscussed OOD result. If the authors can convincingly address these during the rebuttal, I would be happy to raise my score to Accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the AC and Reviewers for the constructive comments. We are encouraged by positive comments: “clinically relevant problem / highly critical clinical issue” (R#3/4), “transformative step, highly novel and timely / forward‑looking” (R#2/4), and “exceptional robustness / convincing ablation / impressive robustness gains” (R#2/3/4). We clarify all concerns below and will revise the manuscript accordingly.
Details about SEER-Trace generation (R#2/3/4): SEER-Trace is generated by DeepSeek-V3.2 with constrained prompts and careful reliability verification. (i) Raw prompts are generated under at eight predefined styles, with each target constrained by aliases, visual priors, and forbidden terms. (ii) The resulting text labels are checked for equivalence to the intended target and then executed by the backbone. The final answer is selected by downstream consistency, ensuring that it is both target-consistent and executable. In the expert audit, only 7 out of 1,120 randomly sampled instances were flagged as abnormal, yielding a 0.63% abnormal rate. This low rate on a random 5% sample supports the reliability of the full SEER-Trace dataset.
Clarification on SEER-Bank framework (R#2/3/4): The SEER-Bank operates on two core phases: construction and retrieval. The construction phase keeps the bank compact and useful by deduplicating skills by dataset, tag, and target, retaining high-gain variants, pruning skills with negative risk-adjusted utility, updating empirical gain from observed marginal improvements, and capping the bank at 5% of the dataset size. The retrieval phase ranks skills using both metadata and evidence-alignment cues (e.g., alias overlap, visual cues), then renders the top candidates as
blocks, from which the VLM selects the most relevant skill in its rationale. In our analysis, ABBR_DISAMBIGUATE is the most frequently retrieved skill, matching SEER's target brittleness. Calculation of latency (R#2/4): On PENGWIN, the frozen-backbone baseline takes 21.18 s/case, while SEER takes 31.37 s/case. This corresponds to an additional 10.19 s per case, which is modest for assisted segmentation settings where robustness to free-text variation is prioritized.
Clarification on multi-view concern (R#2): To achieve efficient and effective inference, we continue to follow current VLM pipelines, leveraging a series of 2D slices as input. To further incorporate 3D information, we adopt a multi-view approach as the input. However, we acknowledge that such a multi-view design is still limited in capturing comprehensive 3D visual information.
Clarification on contribution framing (R#3): We will explicitly position SEER as a reasoning-driven framework in the revised manuscript, and we will further evaluate its performance across additional backbones.
Clarification on evaluation scope (R#3): We have conducted evaluations on two external datasets across different modalities (MRI and CT) and organs (brain and pelvis). Furthermore, we will evaluate our SEER framework on more diverse datasets.
Clarification on OOD wording (R#3): We clarify that our original OOD statement refers to SEER’s reasoning supervision and target coverage, rather than to whether the frozen VoxTell backbone has seen related target anatomies during its own pretraining.
Clarification on higher OOD Dice scores (R#4): We clarify that the magnitude of the Dice score is primarily determined by the size of the segmented ROI, rather than the intrinsic difficulty of the task. Specifically, unlike the small, heterogeneous brain metastases, pelvic bones are large and contrast-stable; thus, they are inherently easier to segment at the voxel level despite being OOD.
Optimize readability (R#3/4): We corrected the notation issues and weakened all overstated claims (e.g. continuous adaptation).
All comments will be addressed in the revised manuscript.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a conceptual novel and interesting framework, and all reviewers agree to accept this paper.
