List of Papers Browse by Subject Areas Author List
Abstract
Language-driven medical image segmentation is critical for clinical workflows, yet existing models struggle with linguistic variability and complex diagnostic queries. These architectures often process semantic and spatial information in isolation, making it difficult to merge abstract clinical reasoning with precise anatomical boundaries. To address these limitations, we introduce MedFuse-Seg, an architecture that combines complex reasoning via Q&A generation with multi-level visual feature injection for prompt-based medical image segmentation. First, we construct Med-ReasonSeg, a large-scale reasoning segmentation dataset consisting of 539,383 image-mask-Q&A triplets and encompassing 9 modalities from 16 datasets. We generate referring and semantic Q&A pairs that are post-processed with a hierarchical LLM-based pipeline to ensure logical fidelity and reduce hallucinations. Second, we propose a Multi-Level Context Fusion mechanism that injects hierarchical features from MedSigLIP (L6-L24) directly into the MedSAM encoder, while leveraging MedGemma reasoning to guide mask decoding. MedFuse-Seg outperforms zero-shot BiomedParse and the fine-tuned LISA-7B by 13.49% and 4.89% in Dice Similarity Coefficient (DSC), and reduces 95th percentile Hausdorff Distance (HD95) by 54.04 and 15.29 pixels, respectively. These results demonstrate that combining reasoning-augmented training data with multi-level feature injection can address semantic and spatial gaps in medical segmentation. Our code and dataset are publicly available at https://github.com/biodatlab/medfuse-seg.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2934_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/biodatlab/medfuse-seg
Link to the Dataset(s)
Med-ReasonSeg dataset: https://huggingface.co/datasets/biodatlab/Med-ReasonSeg
Our curated reasoning segmentation dataset aggregated from 16 open-access sources. Detailed attribution and downstream URLs for all 16 source datasets are fully documented within this Hugging Face repository.
BibTex
@InProceedings{LimKee_MedFuseSeg_MICCAI2026,
author = { Limaroon, Keetawan AND Chiewhawan, Monrada AND Timklaypachara, Watcharapong AND Vateekul, Peerapon AND Achakulvisut, Titipat},
title = { { MedFuse-Seg: Multi-level Visual and Semantic Context Fusion for Segmentation-Based Medical Reasoning } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper makes two main contributions. First, it introduces Med-ReasonSeg, a large-scale medical reasoning segmentation dataset with 539,383 image-mask-Q&A triplets constructed from 16 datasets and 9 imaging modalities, with LLM-based generation and verification of referring and semantic queries. Second, it proposes MedFuse-Seg, a language-driven medical segmentation framework that combines MedSigLIP, MedGemma-4B, and MedSAM through multi-level visual feature fusion and reasoning-guided mask decoding.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1)The paper addresses an important problem and introduces a large-scale reasoning segmentation dataset with broad modality coverage. (2)The proposed framework is well designed and supported by baseline comparisons, qualitative results, and ablation studies.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
(1)The methodological novelty is limited because the framework mainly combines existing components and established ideas. (2)The experimental validation is not fully convincing because the main benchmark is private and some baselines are only evaluated in the zero-shot setting. (3)There is no comparison with the latest inferential segmentation work, such as Medisee (ACM MM), etc.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the paper addresses a relevant problem and presents a technically solid framework, the methodological novelty is limited and the experimental validation is not fully convincing. In particular, the approach mainly combines existing components and ideas, while the main evaluation is centered on a self-constructed benchmark with some baselines only tested in the zero-shot setting. Overall, the work is promising, but the current evidence is not yet strong enough for acceptance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper’s two main contributions are: (1) a 2D medical reasoning segmentation dataset (engineering effort), and (2) a LISA-style modification that injects text semantics into the image encoder, improving accuracy (method novelty).
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths: 1.The authors leveraged Gemini 2.5-Pro to create a 539K-sample dataset (medical images, texts, masks, Q&A pairs), representing an engineering contribution. 2.The proposed “Feature Fusion” method yields noticeable improvements (e.g., +5% DSC over baseline LISA in Tables 1 and 2), indicating a certain level of contribution to segmentation performance.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Weakness: 1.Unclear primary contribution and imbalanced presentation. The paper presents two contributions (dataset and model) but focuses disproportionately on dataset description. This results in limited space for method explanation and essential experiments, making the primary innovation unclear and insufficiently supported. 2.The “[SEG] token positioning” ablation shows negligible differences (0.22% DSC, 1.08 HD95), likely due to training noise. The [SEG] token is generated autoregressively based on the full preceding context, regardless of its absolute position. The authors’ claim that placing [SEG] at the end “allows the model to aggregate full semantic context before segmentation” is not well-supported—as shown by the original LISA output “it is [SEG]”, where no such aggregation occurs before the [SEG] token. This experiment causes confusion. 3.Although the proposed Feature Fusion method yields substantial quantitative gains (+5% DSC, –13.91 HD95), the paper does not provide sufficient analysis to explain why these improvements occur. For instance: How does the feature distribution change after fusion (e.g., feature discriminability, class separation)? Is the gain primarily due to better text reasoning accuracy, or does the fusion more directly improve [SEG] token precision (similar to the decomposition analysis in [20])? Without such analysis, the source of improvement remains unclear, limiting the scientific insight and generalizability of the method.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Promising performance gains and dataset, but major revisions needed on contribution clarity, ablation, and missing analysis.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
This work shows promising performance gains and dataset contributions. Further work is required on contribution clarity, ablation, and a deeper analysis beyond accuracy — such as on feature distribution and feature visualization.
Review #3
- Please describe the contribution of the paper
Experimental results demonstrate that combining inference-enhanced training data with multi-level feature injection can bridge the semantic and spatial gaps in medical segmentation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
3.Although the authors’ proposed MedFuse-Seg integrates three powerful foundational models (MedSigLIP, MedGemma, and MedSAM), it achieves effective parameter efficiency by fine-tuning only the linear layers (r=64) using LoRA technology and freezing the main backbone network
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The core framework of this model relies heavily on existing research paradigms, and its discussion of originality lacks depth. Although the paper proposes “multi-level context fusion,” this is essentially just a variation of the common practice of 1x1 convolution for dimensionality reduction and feature concatenation. The application of this method in the medical field appears more like an empirical engineering improvement than a fundamental theoretical breakthrough. By introducing special [SEG] tokens and projecting their hidden states as segmentation prompts, this approach directly adopts the core idea of LISA (2024). 2.The paper decomposes 3D volumetric data into separate 2D slices for processing. This approach completely ignores the anatomical continuity and spatial context of lesions along the axial (Z-axis) direction. In real-world clinical diagnosis, physicians rely on continuous slices to determine lesion boundaries. Compared to existing general-purpose 3D medical segmentation models ，such as SegVol(2024), MedFuse-Seg may suffer from inter-slice inconsistencies when handling complex volumetric lesions. 3.Although the paper employs a two-stage model screening process, it essentially remains a self-validation exercise within a single model family (the Gemini series). An ideal workflow should incorporate cross-model cross-validation, utilizing large language models with different architectures (such as GPT-4o or Claude 3.5) for multidimensional comparison to identify inherent biases in specific models. The paper did not establish a “gold standard” validation set supported by clinicians. In serious medical contexts, relying solely on model self-assessment is insufficient.
[1] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9579–9589 (2024)
[2] Du, Y., Bai, F., Huang, T., & Zhao, B. (2023). SegVol: Universal and interactive volumetric medical image segmentation. arXiv preprint arXiv:2311.13385
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this work has limitations in terms of 3D spatial modeling and raw data validation, its contributions remain of significant value to the medical imaging community, given its exploration of the cutting-edge field of inference-based segmentation, the production of a large-scale dataset, and the excellent performance demonstrated in multimodal benchmarks. The work would be more convincing if the authors could further discuss in their response how to incorporate spatial consistency constraints or provide a small-scale manual quality assessment report.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal effectively clarifies the paper’s practical contribution as a medically contextualized reasoning-segmentation framework rather than a fundamentally novel architectural paradigm. The additional ablation studies and complexity-specific analyses provide stronger evidence that the proposed multi-level fusion improves boundary precision, especially for anatomically challenging cases. The clarification regarding expert-reviewed samples also partially alleviates concerns about dataset quality control.
However, concerns regarding the limited architectural novelty and the lack of explicit volumetric modeling remain valid. In particular, the 2D slice-based formulation may still suffer from inter-slice inconsistency in complex 3D lesions. Nevertheless, given the scale of the dataset construction effort, the strong empirical performance, and the practical relevance of reasoning-guided medical segmentation, the rebuttal substantially improves confidence in the work.
Author Feedback
We thank all reviewers for their constructive feedback (R1, R2, R3). We address the major comments below.
[Novelty: R1.1, R2.1, R3.1] Our contribution goes beyond simply integrating existing components. We introduce a fusion module that aligns multi-scale VLM features (L6, L12, L18, and L24) with MedSAM’s encoder via point-wise convolutions and refines boundaries through cascaded ConvNeXt blocks following pyramidal fusion principles. This direct visual context sharing between reasoning and segmentation is particularly important for medical images with irregular lesion boundaries. Ablation studies further confirm that the proposed fusion mechanism is the key driver of improvement, increasing performance from 0.7373 DSC (without feature fusion) to 0.7873 DSC (with feature fusion). Beyond architecture, our work prioritizes medical contextualization in both model selection and dataset construction. We argue that an integrated approach spanning task definition, synthetic data generation, and quality control is essential for medical reasoning datasets where accuracy is key.
[Baseline Selection: R1.2, R1.3] To our knowledge, few models specifically address medical reasoning segmentation. The baselines selected here represent foundation models for comparable tasks: BiomedParse, which performs reasoning segmentation under fixed prompt templates; MedSAM, a promptable segmentation model adapted for medical imaging; G-DINO with MedSAM, which employs detection-guided segmentation; and LISA-7B, a generalist reasoning segmentation model. Notably, our model maintains superior performance with a 4.89% improvement in DSC over LISA-7B even after fine-tuning. We acknowledge MediSee as a relevant concurrent work. However, MediSee was recently published, and its training code was just released in November 2025, leaving limited time before the MICCAI 2026 submission deadline (February 12, 2026) to fairly reproduce and benchmark the method. We plan to include more comprehensive comparisons in future work and will clarify these limitations more thoroughly in the revised manuscript.
[SEG Positioning: R2.2] We agree that DSC gain (+0.22%) is modest and will moderate our claim. We hypothesized that [SEG] positioning matters because it determines how much reasoning context the model can condition on before initiating segmentation. However, both metrics are statistically robust (p < 0.001) across test samples. We present this as validation of our hypothesis, motivating future work on query formulations.
[Gains: R2.3] We calculated the gain and found that complex targets improve substantially (LIDC +17.3%, MSD-colon +11.7%, breast-CDD +10.8% DSC) while single-structure targets improve minimally (MSD-heart +1.7%, MSD-spleen +1.7%). HD95 reinforces this pattern, where complex cases improve by over 30px (up to 65.7px in LIDC), while simple cases improve by 2px. This proves fusion improves boundary precision, not just reasoning. Ablations confirm L6/12 localizes and L18/24 refines.
[2D vs. 3D: R3.2] While 3D continuity is ideal, our current focus is on demonstrating that multi-context-level reasoning improves segmentation performance relative to foundation models operating in the same 2D setting. Although SegVol represents foundational work in volumetric text-prompted segmentation, it falls outside our scope, as it lacks referring expression capabilities entirely. We leave the 3D extension for future work.
[Dataset Validation: R3.3] We prevent hallucination at the source by grounding the Q&A in real visual evidence. We designed prompts that force the outputs to describe only the provided mask, minimizing model bias during Q&A generation. For verification, we applied a medical-specific checklist to check for anatomical or biological errors. To address R3, domain experts reviewed sampled generated cases (Sec 4.2) across all organs and modalities to confirm clinical accuracy before fine-tuning. We will release all prompts and datasets upon acceptance.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All three reviewers raise concerns about limited methodological novelty — the framework largely integrates existing components (MedSigLIP, MedGemma, MedSAM) and follows the LISA paradigm. The authors should clarify the core technical contribution beyond engineering integration. The experimental validation also needs strengthening: R1 notes that the main benchmark is self-constructed and several baselines are only evaluated in the zero-shot setting, with missing comparison to recent reasoning-segmentation work (e.g., Medisee). These points should be addressed in the rebuttal. R2 raises an important methodological concern that the reported +5% DSC gain from Feature Fusion lacks supporting analysis of why it works (e.g., feature discriminability, decomposition of [SEG] precision vs. text-reasoning gain), and that the [SEG] positioning ablation (0.22% DSC) is not convincing. Deeper analysis of the fusion mechanism would substantially strengthen the paper. R3 additionally raises a valid concern that decomposing 3D volumes into independent 2D slices ignores axial anatomical continuity (cf. SegVol); the authors should discuss inter-slice consistency.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The rebuttal addresses these reasonably well: it articulates the fusion module’s design (multi-scale VLM–MedSAM alignment with cascaded ConvNeXt refinement) and, importantly, decomposes the gain by target complexity to show the improvement stems from boundary precision (e.g., LIDC +17.3% DSC, HD95 improving by 30px+) rather than reasoning alone. The authors also reasonably justify the omission of MediSee as concurrent work (code released Nov 2025), moderate the [SEG]-positioning claim, and clarify the expert clinical validation of the dataset. While the novelty is more integrative than fundamental, the contribution is solid and the empirical analysis is convincing.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Based on the outcome of the rebuttal phase, two reviewers assigned an “Accept” score, while the third reviewer did not provide a final post-rebuttal assessment, maintaining a “Weak Reject” stance. However, it is worth noting that this reviewer emphasized that “the work is promising” and “addresses a relevant problem and presents a technically solid framework.” The rebuttal also served to clarify “the paper’s practical contribution as a medically contextualized reasoning-segmentation framework rather than a fundamentally novel architectural paradigm” (R3). Consequently, the overall sentiment of the scores was positive, and the final decision was “Accept.”
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper addresses an emerging problem in medical reasoning segmentation, contributing both a large-scale Med-ReasonSeg dataset and a practical multi-level visual-semantic fusion framework. Although the concerns remain about 2D slice-based processing and deeper fusion analysis, the rebuttal clarified the source of performance gains, the dataset validation process, and baseline selection. Overall, the strong empirical performance and dataset contribution support acceptance.