Abstract

The fine-tuned Segment Anything Model has shown strong potential for medical image segmentation, yet it remains class agnostic and cannot provide reliable categorical information alongside the predicted mask. This limits interpretability and increases the risk of selecting anatomically implausible regions when boundaries are ambiguous. Meanwhile, existing medical frameworks often treat segmentation and classification as separate processes, which weakens feature interaction and reduces robustness across diverse organs, lesions, and imaging modalities. We present BiMI, a mutually promoted framework that integrates a fine-tuned Segment Anything Model with a hierarchical classification branch. To regularize fine-grained recognition under data scarcity, BiMI introduces a Coarse2Fine Matrix that models hierarchical dependencies and enforces semantic consistency between coarse and fine predictions. To strengthen feature interaction between recognition and segmentation, BiMI further adopts a Global2local Feature Enhancement module that aligns global context with local spatial cues. In addition, BiMI incorporates a Text Guidance module that expands the predicted coarse and fine labels into a compact structured description using a frozen language model, and encodes it into a semantic embedding with a frozen text encoder. The embedding conditions structure-aware candidate selection to rank multi-mask hypotheses within the box prompt, improving anatomical plausibility and segmentation stability. Experiments on six public medical datasets covering CT, MRI, dermoscopy, and endoscopy demonstrate that BiMI consistently improves hierarchical classification and achieves competitive segmentation performance against strong U-Net and SAM-based baselines, with clear gains on several challenging benchmarks.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5545_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{DenZij_Mutually_MICCAI2026,
        author = { Deng, Zijian AND Yao, Xiaoli AND Zhao, Qijun AND Shahzadi, Iqra AND Yang, Hui},
        title = { { Mutually Promoted Medical Image Segmentation and Classification via Prompt-Driven Interaction } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16883},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper presents a framework integrating a fine-tuned Segment Anything Model (SAM) with hierarchical classification, bridging the gap between image segmentation and fine grained medical recognition. It addresses SAM’s class-agnostic limitations, achieving excellent classification and segmentation performance against U-Net and SAM-based baselines.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper provides a novel integrated SAM-classification framework (C2FM, GFE, SACS modules) with well structured methodology and ablation. It demonstrates strong validation with benchmark methods on 6 datasets, and shows competitive performance results. Overall, there is clinical potential as the method potentially improves interpretability.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Limitations include the reliance on manual box prompts, rather than a fully automated and scalable clinical workflow. There is incomplete evaluation (eg metrics limited to DSC/IOU), and the classification results are somewhat weak on key datasets (such as AbdomenCT1K). SACS is presented as one of the major contributions but not isolated in the ablation (bundled within TG). There are also some missing elements for reproducibility, e. g. the LLM is never identified or cited.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The idea is well-motivated and relevant. The evaluation is broad, covering six public datasets across different imaging modalities (CT, MRI, dermoscopy, endoscopy). Performance results are competitive. There are some weaknesses and aspects requiring clarification but do not overall undermine the core contribution.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The rebuttal clarifies some but not entirely all of the concerns.



Review #2

  • Please describe the contribution of the paper

    This paper proposes a mutually reinforcing framework called BiMI, which integrates a prompt-based segmentation model built on the Segment Anything Model (SAM) with a hierarchical classification branch for medical image analysis. BiMI introduces three core components: (1) a Coarse-to-Fine Matrix (C2FM) to model hierarchical dependencies and regularize fine-grained recognition under data-scarce conditions; (2) a Global-to-Local Feature Enhancement (GFE) module to align the global context from the SAM encoder with local spatial cues; and (3) a Structure-Aware Candidate Selection (SACS) module, which leverages a frozen language model and text encoder to generate semantic embeddings, and then uses these embeddings to rank multiple mask hypotheses within bounding-box prompts. Extensive experiments on six public medical datasets, demonstrate that, compared with strong baseline models, BiMI achieves competitive segmentation performance while substantially improving the accuracy of hierarchical classification.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Innovative mutually reinforcing framework: By explicitly modeling the bidirectional interaction between segmentation and hierarchical classification, the proposed BiMI framework effectively addresses the issue of semantic inconsistency and improves the anatomical plausibility of medical image segmentation.

    2.Technical contributions: The Coarse-to-Fine Matrix (C2FM) provides a soft regularization mechanism to alleviate data scarcity in fine-grained classification. The Global-to-Local Feature Enhancement (GFE) module successfully aligns global and local features. In addition, the Structure-Aware Candidate Selection (SACS) module cleverly leverages text-guided semantic priors to resolve the inherent multi-mask ambiguity of SAM.

    3.Comprehensive experimental validation: The authors conduct extensive experiments on six diverse public datasets covering multiple imaging modalities (CT, MRI, dermoscopy, and endoscopy) and targets (organs and lesions). The results demonstrate that BiMI outperforms both U-Net-based baselines and recent SAM-based baselines on both segmentation and classification tasks.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Dependence on manual bounding-box prompts: The current framework relies on manually provided bounding-box prompts for localization, which limits its feasibility in fully automatic clinical applications. Although the authors mention adaptive bounding-box refinement as a direction for future work, they should more carefully discuss the practical clinical relevance of manual prompting compared with fully automatic segmentation pipelines, and possibly consider a user study to justify this design choice.

    2.Computational overhead and complexity: Integrating a frozen large language model (LLM), a text encoder (CLIP), and the SAM architecture introduces substantial computational complexity. The paper lacks a detailed analysis of inference time, memory consumption, and overall computational cost in comparison with simpler baseline models such as nnU-Net or standard SAM.

    3.Lack of discussion of failure cases: Although the quantitative results are strong, the paper would benefit from a more in-depth qualitative analysis of failure cases.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please refer to the summary and the weakness sections.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    My concerns are mostly addressed, and I am still positive with this work after rebuttal. I am happy to recommend an acceptance.



Review #3

  • Please describe the contribution of the paper

    This paper proposes BiMI, which is a medical imaging framework that integrates cue-based segmentation with hierarchical classification. The authors construct the C2FM to establish a transition matrix between coarse and fine classes, and use KL divergence to constrain fine class predictions to conform to the coarse class distribution. The GFE contains several designs: (a) SA, (b) FA, (c) Auxiliary Binary Classifier, (d) Cross-Attention Fusion. The SACS uses text to guide the mask selection.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.This method achieves a unified framework for multi-task collaborative optimization and cross-modal knowledge fusion through an innovative bidirectional mutually reinforcing architecture of segmentation and classification, a hierarchical semantic constraint C2FM module, and text-guided anatomical rationality verification. 2.The proposed feature alignment mechanism retains global discriminative power, avoiding over-alignment. Cross-attention enables dynamic information interaction. 3.The experimental results demonstrate the effectiveness of the proposed method.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The manuscript does not follow the MICCAI26 format. 2.The major limitation of this methos is the requirement of manual provision of bounding boxes to locate the target region. The paper acknowledges this in the Conclusion but does not provide a solution. 3.Insufficient validation of the necessity of the two-stage training strategy. Why freeze C2FM training for 50 epochs before joint training in the second stage? Has end-to-end training been attempted? What is the performance difference? 4.The compared SAM-based methods (MedSAM, SAM-Med2D, etc.) are all from 2023-2024.Lack of comparison with stronger baselines like SAM2 and MedSAM2.5.Insufficient interpretability of the SACS module. Definitions of descriptors (compactness, solidity, connectivity, gradient entropy) are not given in the text. 6.As can be seen from Table 3, all performance aspects decreased after adding GFE. So what is the point of adding this module?

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (2) Reject — should be rejected, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    As described in the weakness.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    Thank you for the authors’ rebuttal. However, my main concerns remain insufficiently addressed. The rebuttal states that the same box-prompt protocol is used as SAM-style baselines, but this does not resolve the limitation of requiring manual localization. The explanation of the two-stage training strategy is still largely empirical. The authors state that end-to-end coupling may lead to unstable training, but no quantitative evidence or ablation is provided to support this claim. The rebuttal regarding SAM2/MedSAM2 is not fully convincing. Even if these models are designed for video or 3D propagation settings, they represent stronger and more recent SAM-family baselines. Without comparison to them, it is difficult to assess the actual competitiveness of the proposed framework. Although the authors clarify that GFE without FA causes degradation, the ablation still shows that the gain of the full design over SA+TG+C2FM is marginal. Moreover, since GFE is presented as an individual innovation module, the fact that it substantially degrades performance unless coupled with another alignment module raises concerns about its intrinsic reliability and robustness. Overall, while the rebuttal clarifies some implementation details, it does not adequately address the core concerns. Therefore, I maintain my negative recommendation.



Author Feedback

We thank the AC and reviewers for their time and constructive feedback. We are encouraged by their acknowledgement of the novelty and the superior results of our work. Below we do our best to resolve the key concerns raised by them. 1.Format. We apologize for the confusing running header. We used the official MICCAI 2026 template without modifying margins, font, spacing, or page limit. We will correct the missing short title. 2.GFE. The performance drop in Table 3 occurs only for GFE without FA (i.e., unaligned global feature injection), not the complete design. This verifies our hypothesis in Sec. 2.2: direct global-local fusion may cause semantic misalignment. SA+TG+C2FM gives 78.8/87.3 IoU/DSC; unaligned GFE drops to 73.9/85.3; adding FA gives the best 79.1/87.7.These results do prove the effectiveness of our method. 3.Training. C2FM is not a trainable branch. It serves as a coarse-to-fine prior for the classifier. According to our experience, direct end-to-end coupling of C2FM and other modules would result in unstable training behavior. We believe this is because early inaccurate class predictions can mislead mask selection, while early coarse predictions also introduce noisy priors for fine classification. Thus, we first stabilize the segmentation-related components and classifier, and then integrate them via C2FM, GFE/FA, and SACS. 4.SAM2/MedSAM2.Our comparison focuses on representative U-Net-based and medical SAM-based baselines under the same 2D slice-wise box-prompt protocol. SAM2/MedSAM2 are mainly designed for image-video or 3D propagation settings; while their native protocol defines different tasks, forcing single-slice inference would not reflect their intended design. As for BiMI, its contribution is the segmentation-classification interaction mechanism, which is orthogonal to the SAM backbone. 5.SACS/TG. Sorry for the unclear notation. In Table 3, TG is actually TG with SACS (or TG/SACS) rather than an independent text-only module: it denotes the text-guided candidate-selection path implemented by SACS. Text embeddings are mapped to structural weights. SACS combines them with mask descriptors and SAM’s IoU prediction to rank candidates and select the final mask. Because text is not injected into the decoder, TG without SACS is undefined in our architecture. Our early trial shows that direct text-feature concatenation did not work in practice, which motivates SACS. In Table 3, the TG on/off comparison corresponds to the SACS-path on/off ablation, improving IoU/DSC from 78.0/86.1 to 78.5/86.9.We will rename TG as TG/SACS. 6.Cost, LLM, and ethics. Our BiMI’s online overhead mainly comes from the classifier, GFE/FA, and SACS candidate-mask scoring. The language step is offline: GPT-4 generates coarse/fine label-pair descriptions once, and CLIP encodes them into a precomputed embedding bank. At inference, BiMI only retrieves the embedding for the predicted label pair and applies descriptor-based ranking to SAM candidates, with no per-image LLM call. Apart from SAM and the classifier, the language-guided SACS overhead is limited to embedding lookup, mask-descriptor computation, and score fusion. The LLM receives no patient images, masks, metadata, or clinical notes, and is not used for diagnosis or treatment decisions. Therefore, its use does not incur patient-data privacy or LLM-driven clinical-decision risks. 7.Prompting and evaluation. We acknowledge that BiMI is not fully automatic. We use the same box-prompt interface as SAM-style baselines, adding no extra localization burden. DSC/IoU were used because they are consistently reported across the six datasets for fair comparison. AbdomenCT1K is difficult for all classifiers due to similar slice-level CT appearances, but BiMI remains best under the same protocol.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Scores are 4/4/2, with two reviewers recommending weak accept and one PhD-student reviewer recommending reject “independent of rebuttal” on the basis of format non-compliance, the GFE-hurts-performance ablation, missing modern baselines, and insufficient training-strategy validation. The paper is borderline-leaning-positive, but several of R3’s concerns — particularly the GFE ablation contradiction and the format compliance — are concrete enough that a credible rebuttal is required to retain the weak accepts. The paper is invited for rebuttal; authors should prioritize (a) a direct, principled explanation of the GFE ablation result that shows performance decreasing when GFE is added, (b) at least one comparison against SAM 2 / MedSAM 2 or a clear justification for omitting them, (c) a properly isolated SACS ablation and an end-to-end vs. two-stage training comparison, (d) computational-cost analysis and identification of the LLM used in SACS, and (e) clarification of the ethics concern flagged by R1.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Reviewers agreed that the bidirectional segmentation-classification framework is novel and well-motivated, the three modules (C2FM, GFE, SACS) are clean, the multi-modality evaluation across six datasets is comprehensive, and the clinical potential for interpretability is meaningful. They raised several concerns: (1) reliance on manual bounding-box prompts limits clinical deployment in fully automatic workflows; (2) computational overhead from integrating LLM + CLIP + SAM is not analyzed; (3) recent stronger SAM-family baselines (SAM2, MedSAM2) are missing; (4) the two-stage training strategy is not validated against end-to-end coupling; (5) SACS is bundled with TG in ablation, making its individual contribution unclear; (6) GFE alone (without FA) degrades performance, raising intrinsic-reliability concerns; (7) manuscript format issues (running header, references vs. bibliography); (8) the LLM is not explicitly identified or cited; and (9) failure case analysis is missing.After rebuttal, the authors clarified that GFE’s degradation occurs only without the FA alignment component (the full design achieves the best 79.1/87.7 IoU/DSC), justified the two-stage training based on instability of direct C2FM end-to-end coupling, argued SAM2/MedSAM2 target different protocols (video/3D propagation) orthogonal to BiMI’s contribution, renamed TG to TG/SACS to clarify the ablation, explained that the LLM/CLIP pipeline is offline (no per-image LLM call, no patient data exposure), and confirmed the box-prompt protocol matches SAM baselines. However, the most critical reviewer maintained that the manual-prompt limitation remains, the two-stage training justification is empirical without quantitative ablation, the SAM2/MedSAM2 absence remains a real comparison gap, and GFE’s dependence on FA raises intrinsic-reliability concerns. Final scores are split (Accept, Accept, Reject). Overall, the AC believes that the merits outweigh the drawbacks and recommend accept.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Reject

  • Please justify your recommendation.

    After rebuttal the reviews stand at two leaning accept and one reject.

    The rebuttal does clarify the format and SACS/TG notation issues and gives a self-consistent account of the GFE+FA design, and the two positive reviewers are satisfied.

    However, the concerns that bear on the paper’s contribution remain. The method is one of many incremental SAM box-prompt adaptations, and the ablation shows the full design improves only marginally over SA+TG+C2FM, with GFE actually degrading performance unless paired with FA — which weakens the case for it as a standalone contribution. Given the limited novelty and marginal gains relative to existing SAM-prompt methods, I lean toward rejection despite the rebuttal’s clarifications, as a borderline case.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Two reviewers recommended acceptance after rebuttal, recognizing the innovative framework and comprehensive validation. The authors’ rebuttal partially addressed R3’s concerns. Although some minor concerns such as relying on manual box prompts and comparison with SAM2 remain, I believe they don’t overweigh the contribution.



back to top