Abstract

Transforming foundation segmentation models from human-prompted tools into auto-promptable annotators is critical for scalable medical data annotation. Current methods commonly depend on external feature matchers or auxiliary networks to automate geometric prompting, but introducing architectural overhead and limiting performance scalability. Although SAM3 natively supports concept segmentation via reusable text prompts, its direct use in medical imaging is hindered by a lack of fine-grained clinical knowledge and the ambiguity of human-written descriptions. In this work, we propose Mask to Concept (M2C), an efficient framework that adapts SAM3 for medical few-shot annotation without external modules, parameter retraining, or manual text engineering. Using only a few labeled images, M2C enables SAM3 to automatically search for transferable visual concepts entirely within its frozen architecture: it initializes a learnable concept embedding, uses it to prompt segmentation, and updates the embedding by gradients of minimizing the concept segmentation error. We further introduce a Hybrid Uncertainty Estimation (HUE) module that calculates the prediction entropy and maps concept predictions back to the box prompts, measuring concept-geometry prompting inconsistency. Highly uncertain samples are flagged actively for human correction, and the corrected masks are then fed back to M2C to continuously search for more precise concept embeddings, forming a self-enhancing annotation loop with minimal expert effort. Experiments on medical segmentation benchmarks show that our method achieves SOTA few-shot segmentation performance and outstanding annotation efficiency, offering a practical and efficient pathway toward scalable medical image labeling. Codes are at https://github.com/Huster-Hq/M2C.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1221_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/Huster-Hq/M2C

Link to the Dataset(s)

N/A

BibTex

@InProceedings{ZhoQua_Mask_MICCAI2026,
        author = { Zhou, Quan AND Zhai, Shaoqing AND Hu, Qiang AND Chen, Jia AND Li, Qiang AND Wang, Zhiwei},
        title = { { Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper introduces Mask to Concept (M2C), a training free framework that turns SAM3 into an auto promptable annotation system for medical image segmentation. The core idea is to learn a concept embedding from only a few labeled images by optimizing it through segmentation feedback, enabling the model to automatically discover transferable visual concepts within a frozen SAM3 backbone. This removes the need for external prompt generators or additional trainable modules, and instead relies on iterative refinement of the concept embedding guided by segmentation error. The method is further extended with a Hybrid Uncertainty Estimation module that identifies uncertain predictions by measuring entropy and concept geometry inconsistency, and uses these cases to trigger human correction. Corrected annotations are then fed back into the system, forming an active, self improving annotation loop aimed at reducing expert labeling effort.

    Overall, the experimental results suggest strong few shot segmentation performance and improved annotation efficiency across medical benchmarks, supporting the practical value of the approach. However, some of the performance and generalization claims may be somewhat overstated, and the strength of the improvements likely depends on the specific benchmarks and baselines used.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The main strengths of the paper are the strong and convincing experimental results, supported by excellent ablation studies and insightful analysis. The empirical evaluation is particularly thorough, with carefully designed experiments that clearly demonstrate the effectiveness of the proposed M2C framework. The annotation efficiency analysis is a highlight of the work: the results show that the hybrid uncertainty estimation not only improves active selection of hard samples but also enables a meaningful early exit strategy for easy samples, which are confidently segmented via concept aligned embeddings before further updates. This design leads to consistently better performance under fixed annotation budgets and clearly outperforms random sampling baselines, while the full system achieves the best annotation quality with minimal human effort. The ablation studies further strengthen the paper by showing that both uncertainty components contribute complementarily, and that combining them is essential for efficiently distinguishing easy and hard samples to reduce annotation cost. In addition, the paper is very well written with a clear and compelling storyline that guides the reader through motivation, method, and results in a coherent way. The qualitative and quantitative results are well aligned and reinforce each other, and the comparisons on annotation efficiency provide a strong practical argument for the system’s usefulness in real medical labeling workflows. Overall, the presentation is strong and contributes significantly to readability and impact.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The main weakness of the paper lies in the limited mathematical grounding and interpretability of the proposed uncertainty formulation. While entropy is a well established and theoretically sound uncertainty measure, the concept geometry prompting inconsistency term is less rigorously motivated, making the final hybrid uncertainty score feel somewhat heuristic despite strong empirical performance. Closely related to this, it remains unclear whether the resulting uncertainty estimates are well calibrated. The paper does not provide a thorough analysis of calibration quality or position the proposed uncertainty formulation within the broader landscape of uncertainty estimation methods in medical imaging. This makes it difficult to assess how the method relates to established probabilistic or Bayesian approaches. In addition, the overall algorithm resembles iterative refinement strategies similar in spirit to expectation maximization, but this connection is not explicitly discussed or formalized. A clearer theoretical framing could improve interpretability and strengthen the methodological contribution. Finally, the dataset selection is somewhat limited, relying on relatively standard and potentially dated benchmarks, with limited diversity in modalities and task settings. Broader evaluation would improve confidence in the generality of the findings. Overall, while the engineering design is strong and the empirical results are convincing, the main limitation is the lack of deeper mathematical rigor, calibration analysis, and broader validation.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (6) Strong Accept — must be accepted due to excellence

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper presents a strong and well executed contribution to training free few shot medical image segmentation and annotation using SAM3.The experimental results are particularly convincing, with a carefully designed evaluation protocol, strong ablation studies, and clear gains in annotation efficiency under limited human labeling budgets. The proposed hybrid uncertainty estimation and early exit mechanism are shown to be highly effective in practice, and the empirical evidence is consistent across settings. In addition, the paper is very well written with a clear storyline that makes both the motivation and technical approach easy to follow.

    The main limitations are relatively minor compared to these strengths. The hybrid uncertainty formulation is only partially theoretically grounded, as the concept geometry prompting inconsistency term is heuristic in nature and its connection to established uncertainty theory is not fully developed. It also remains unclear whether the resulting uncertainty estimates are well calibrated, and a deeper comparison to standard uncertainty estimation frameworks in medical imaging would strengthen the positioning of the method. Finally, the dataset selection is somewhat limited in diversity, which slightly restricts conclusions about broad generalization.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    While I may have emphasized the strengths in my initial review, the rebuttal did not change my overall assessment. The paper presents a solid and well-executed contribution.



Review #2

  • Please describe the contribution of the paper

    In this paper, the authors introduce a new approach to adapt pretrained segmentation models for medical images in order to solve the medical few-shot labeling challenge. SAM-3 pretrained model is extended with learned concept embeddings. A hybrid uncertainty estimation is introduced as well to be able to carry out active learning. The authors demonstrate the superiority of this approach by achieving SOTA results in Kvasir-SEG and ISIC-2017.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    (*) The considered task is practically relevant. With the emergence of strong text promptable feature extractors like SAM3, the idea of repurposing those model for domain specific medical applications is timely. The authors identify the SAM3 text prompting shortcomings for fine-grained medical concepts.

    (*) The training is very efficient computationally since it solely requires learning the concept embedding. This is a major advantage compared to methods requiring auxiliary networks or feature matchers.

    (*) The combination of prediction entropy and concept-geometry inconsistency is well-motivated and also technically sound. Empirically, it shows complementary benefits for the overall quality.

    (*) strong empirical results: regardless of any other consideration, the combination shows a superior performance to any other active learning approach. it is very likely that this approach also generalizes to other domains. Given the publicly available code, testing this approach would be a low-hanging fruit.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    (*) The concept embedding search is per se not new. It is functionally identical to prompt tuning or textual inversion (Gal et al., 2022; Zhou et al., 2022), both cited by the authors. Extending pretrained SAM3 with this concept remains limited in novelty.

    (*) Claiming a training free method is not correct. The authors still have to train the concept encoding. it is definitely a minor fraction with very low computational implications but the authors are deliberately overselling their method.

    (*) Setting the threshold arbitrarily to 0.6 without an appropriate sensitivity analysis raises concerns about stability.

    (*) Critical evaluation: It is important to have a baseline using SAM3 with optimized text prompts. either manually or automatically. As of now it’s unclear whether the learnable embedding E is actually superior to simply finding better text descriptions.

    (*) Unfair comparison: SAM3 vs. SAM-H baselines. SAM3 is a larger and stronger than the original SAM model. As of now the contribution of the M2C is questionable since the performance gains may reflect just the stronger base model.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Based on the limited novelty and evaluation concerns explained in the weaknesses, I tend to no accept this paper.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    Strengths of the rebuttal include: 1-fixing some of the missing ablations/baselines, 2-clarity in explaining the annotation process, 3-partially addressing fairness issues.

    Weaknesses of the rebuttal include: 1-Theory supporting the notion of uncertainty, 2-Calibration issues, 3-Originality compared to prompt tuning/textual inversion,

    It is important that the “training-free” claim be addressed, as it is included in the title of the paper.

    In general, the paper is extremely practical while theoretically vulnerable.



Review #3

  • Please describe the contribution of the paper

    The paper describes an approach to reduce the dependency of the Segment Anything Model on geometric or textual priors/inputs.

    The proposed approach relies on segmentation based on concepts derived from text prompts, as introduced in SAM3.The method aims to reduce the annotation burden.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The proposed approach requires only a few parameters to be trained.

    The evaluation is well conducted, and the ablation study is useful.

    The performance of the proposed approach is strong, requiring only a small number of annotations.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The paper claims that no external modules or additional parameters are required, which is somewhat misleading. In practice, the proposed approach relies on training concept embeddings. Even if this approach requires only a few examples, as suggested by the results, some training is still necessary.

    The approach is evaluated on relatively simple segmentation tasks. For a fair comparison, it would be interesting to compare it with a baseline model such as nnU-Net, or a similar method, under the same data conditions. In addition, Figure 3 is unclear. Why do human annotations start at 250 and 180, respectively? These values correspond to around 30% of the ISIC dataset, and basic active learning approaches might achieve similar performance with the same amount of data.

    The paper does not clearly explain how human annotation is simulated. For images with low confidence, are the annotations applied to the entire image (based on ground truth), or only to the low-confidence regions? Evaluating the number of corrected pixels could be an interesting additional metric.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The proposed solution seems efficient, requiring only few annotation to perform. Some technical details are lacking to fully understand.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors addressed my questions about data efficiency and seem to have also addressed the concerns raised by the other reviewers.



Author Feedback

We sincerely thank all reviewers for constructive comments and for recognizing the ‘practical relevance’(R2), ‘efficient computation’ (R2,R3), ‘superior performance’ (R1,R2,R3), ‘thorough/well evaluation’ (R1,R3). Below we address the specific concerns:

R2(Q2)&R3(Q1): We agree that “training-free” may be interpreted too broadly. Our intended meaning is that M2C requires neither conventional few-shot segmentation (FSS) training nor fine-tuning SAM3.Thus, it is more precisely described as FSS-pretraining-free or SAM3-retraining-free, rather than strictly “training-free”. We appreciate the reviewers’ comments and will revise the wording to avoid misleading.

R1: Q1&Q2: U_CGPI is indeed not derived as a formal Bayesian uncertainty estimator. However, we would like to clarify that many effective uncertainty criteria in practical annotation and active learning systems are heuristic or task-driven not strictly probabilistic. We evaluated calibration quality through an uncertainty-error correlation analysis using 1−Dice as the real error. Spearman correlations for U_ENT,U_CGPI,and U_HU are 0.76,0.73,and 0.81,respectively.The results show that U_CGPI is well calibrated with real errors, while the hybrid score provides the most reliable uncertainty ranking. Q3: Thanks for this insightful observation. Our system indeed shares a conceptual connection with EM. The “E-step” infers latent variables (predicting pseudo-masks) using the current concept embedding, while the “M-step” optimizes it. We agree that EM perspective enhances our theoretical foundation and will include it in the revision. Q4: Due to space constraints, this work evaluates two representative datasets with different modalities and lesions. We acknowledge this limitation and will verify on broader datasets in future work.

R2: Q1: Although concept embedding search plays a similar role to prior works, our novelty lies in bridging SAM3’s text-centric design and the lack of fine-grained medical textual priors using few-shot samples. This simple pipeline achieves strong performance and generalization compared with more complex methods, suggesting a promising direction. More importantly, M2C is not an isolated embedding search algorithm; its core contribution is coupling this mechanism with HUE to form an efficient closed-loop annotation system. Q3: The 0.6 threshold was empirically selected, not arbitrary. On Kvasir-SEG, lowering it to 0.5 increases 0.26× manual cost for only a 0.5% Dice gain, whereas raising it to 0.7 reduces 0.34× cost but drops Dice by 5.3%. Thus, 0.6 is a cost-effective trade-off, and the system is relatively insensitive to moderate variations around this threshold. SOTA results on two datasets with different modalities and lesion types further support generalizability. Q4: We further tested 10 manual and LLM-generated text prompts.The best Dices were only 0.24(Kvasir-SEG) and 0.38(ISIC-2017),confirming SAM3’s limited medical text understanding, aligning with recent work [1]. [1]MedSAM3:Delving into Segment Anything with Medical Concepts Q5: Existing methods mainly rely on SAM’s geometric prompting capability, and this improvement from SAM to SAM3 is limited. After replacing their backbones with SAM3, M2C remains superior: Kvasir 1-shot: ProtoSAM(73.7), Matcher(71.5), SPFS-SAM(71.1), MAUP(48.2), Ours(76.3); ISIC 1-shot: ProtoSAM(68.6), Matcher(57.9), SPFS-SAM(56.3), MAUP(56.5), Ours(75.9).

R3: Q2: On Kvasir-SEG, nnU-Net trained with 1/5/10 annotations obtains Dice of 25.8/34.6/42.6, respectively, significantly lower than M2C. Detailed annotation-efficiency curves are provided in the code link. It is noted that the human annotation starts from x=0, not x=250 or 180 in Fig.3.The bottom region without model-performance curves is cropped only for space efficiency. Q3: For reliability, we use the provided GT masks to simulate human-corrected annotations. The total corrected pixels for each method in Fig.3(a): M2C(15.2M), SPFS-SAM(22.4M), Multiverseg(51.1M).




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper constructs Mask to Concept, a framework designed to adapt SAM2/3 for few-shot medical image segmentation by optimizing concept embeddings to automate the prompting process. The strengths of the work lie in its high computational efficiency, strong empirical performance across multiple benchmarks, and a well-motivated hybrid uncertainty estimation module that effectively reduces manual annotation effort. However, reviewers have raised concerns regarding the “training-free” claim given the optimization of embeddings, the lack of rigorous mathematical grounding for the uncertainty formulation, and the need for fairer comparisons against optimized text prompts or stronger baselines like nnU-Net. Additionally, further clarification is required regarding the simulation of human corrections and the sensitivity of the chosen hyperparameters. With average scores (6, 3, 4) leaning toward a rebuttal to address these methodological clarifications and validation concerns.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Based on the outcome of the rebuttal process, two reviewers assigned an “Accept” score, while one reviewer assigned a “Reject.” Notably, although Reviewer 2 recommended rejection, he also emphasized that “the paper is extremely practical”. The authors should clearly clarify the “training-free” claim in the camera-ready version. Given that the overall sentiment is positive, the inclination is to “Accept” this paper.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Reject

  • Please justify your recommendation.

    The paper proposes to adapt SAM3 for few-shot medical annotation by optimizing a concept embedding from a few labeled masks and using a hybrid uncertainty score for human-in-the-loop sample selection. While the idea is potentially useful, the current evidence is not sufficient for acceptance.

    My main concern is that the claimed contribution is not convincingly isolated. The method is described as “training-free,” but it still performs gradient-based optimization of a learnable embedding using labeled masks, which is closer to test-time prompt/embedding tuning than a truly training-free method. The novelty over existing prompt tuning, textual inversion, and few-shot SAM personalization methods is not clearly established.

    The experimental comparison is also problematic. M2C uses SAM3, whereas the SAM-based baselines use SAM-H, so the reported gains may largely reflect the stronger backbone rather than the proposed concept-search mechanism. The paper lacks essential controls, such as direct SAM3 text prompting, SAM3 with box/point prompts, SAM3 with naive learned prompts, and variants that isolate the effect of concept embedding optimization from the effect of the backbone.

    The annotation-efficiency evaluation is not fully convincing. Human corrections are simulated with ground-truth masks, and the overall quality metric includes human-labeled samples, which can inflate the apparent efficiency. The uncertainty module uses heuristic components and a fixed threshold, but there is no calibration analysis or robust validation that HUE reliably identifies truly difficult or low-quality cases across datasets.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents a framework for few-shot medical image annotation with SAM3 by optimizing concept embeddings and using hybrid uncertainty estimation to reduce manual annotation effort. Although concerns remain about the “training-free” wording and theoretical grounding of uncertainty, the rebuttal clarified these issues. Overall, the work is empirically strong and sufficiently valuable for acceptance.



back to top