Abstract

In-Context Learning has emerged as a promising paradigm for automated medical image segmentation, enabling foundation models like SegGPT to generalize to new tasks using annotated support examples. However, existing approaches typically rely on retrieving raw images as prompts, which suffer from two critical limitations: the inherent noise in raw medical data often misleads the model, and the repetitive retrieval process incurs severe computational latency. To address these challenges, we propose a novel framework, Learnable In-Context Templates, which shifts the paradigm from sample retrieval to template learning. We introduce an In-Context Distillation strategy to optimize a compact set of learnable templates, condensing the representative lesion patterns of a target dataset. By distilling essential semantic features and filtering out raw image noise, these refined templates provide a cleaner, more concentrated support set. Extensive experiments on six datasets demonstrate that our method consistently outperforms state-of-the-art ICL baselines. Notably, our approach shows broad adaptability across diverse medical modalities and achieves a substantial order-of-magnitude reduction in inference latency, offering an efficient solution for clinical deployment. Code will be available at: \url{https://github.com/MembrAI/In-Context-Templates}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4183_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/MembrAI/In-Context-Templates

Link to the Dataset(s)

ISIC2018 dataset: https://challenge.isic-archive.com/data/#2018 TNUI2021 dataset: https://tnui2021.github.io CVC-ColonDB dataset: http://vi.cvc.uab.es/colon-qa/cvccolondb/ BUSI dataset: https://data.mendeley.com/datasets/wmy84gzngw/1

BibTex

@InProceedings{BaoXue_Learnable_MICCAI2026,
        author = { Bao, Xueqi AND Zheng, Chang AND Zhang, Chengwen AND Lao, Qicheng},
        title = { { Learnable In-Context Templates for Medical Image Segmentation via In-Context Learning } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16883},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper describes an approach that leans templates instead of retrieving samples for in-context learning.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    There is some novelty in learning a template and combining knowledge with learned data It is good that publicly available data sets are used for the evaluation results look good compared to the presented baselines

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    overall the Dice scores still look less than optimal for most data sets part of the formatting is not in LNCS style novelty is rather incremental results are not easy to reproduce because nit too many details are given

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    it would be good to remove the grey background from figure 1 as it is not optimal for reading and shaped rather be white Table 2 ynd 3 are not in LNCS format, as they are net to each other and this should not be the case

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    There is some limited novelty and results look good compare to the given baselines. Still scores in DICE do not look optimal and it is hard for me to reproduce the results even though public data sets are being used

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    It remains a borderline paper to me but I think that the rebuttal help a bit to me to better judge this, so it shifts from weak reject to weak accept.



Review #2

  • Please describe the contribution of the paper

    The paper replaces query-time retrieval of raw support examples with a small set of Learnable In-Context Templates. Concretely, it clusters training images in encoder space, initializes one template per cluster from a centroid image-mask pair, and then optimizes each template with a frozen ICL model (In-Context Template Distillation). In inference, the optimized templates are used jointly as the support set, allowing the model to leverage a compact and reusable set of dataset-level semantic priors rather than query-specific retrieved instances. Empirical results from 6 benchmarks have shown that the method consistently and significantly improved SegGPT and UniverSeg.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper presents a clear methodological contribution: it replaces query-time support retrieval with a compact set of learnable image-mask templates, which is a well-defined and practically motivated design choice.
    • The proposed pipeline is coherent and easy to follow. It is model-agnostic in presentation and is validated on two distinct ICL backbones, SegGPT and UniverSeg.
    • The empirical results are consistently positive across six datasets, and the efficiency analysis supports a meaningful practical benefit over retrieval-based prompting.
    • The ablation studies help validate the proposed design.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The main comparison relies on relatively early retrieval-based baselines, while more recent medical ICL methods such as Iris [1], Tyche-IS [2], and ICL-SAM [3] substantially broaden the design space. Empirical evidence showing whether Learnable In-Context Templates improves over these stronger and more recent alternatives, or transfers effectively beyond the two backbones considered here, would be needed to support the paper’s broader claims.
    • The abstract states that “our approach achieves superior robustness against domain shifts and reduces inference latency by two orders of magnitude,” but the current experimental section does not fully substantiate either claim. In particular, the paper does not present a dedicated domain-shift evaluation, and the latency results appear closer to an order-of-magnitude improvement instead.
    • The proposed framework likely shifts computation from inference to training, which can be a reasonable trade-off in practice. That said, the paper does not report the overhead cost of clustering and template optimization, so the full computational trade-off remains unclear.

    [1] Gao, Y., Liu, D., Li, Z., Li, Y., Chen, D., Zhou, M., & Metaxas, D. N. (2025). Show and segment: Universal medical image segmentation via in-context learning. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 20830-20840). [2] Rakic, M., Wong, H. E., Ortiz, J. J. G., Cimini, B. A., Guttag, J. V., & Dalca, A. V. (2024). Tyche: Stochastic in-context learning for medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11159-11173). [3] Hu, J., Shang, Y., Yang, Y., Guo, X., Peng, H., & Ma, T. (2024, December). ICL-SAM: Synergizing in-context learning model and SAM in medical image segmentation. In Medical Imaging with Deep Learning.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper introduces a clean and practically motivated alternative to retrieval-based in-context learning by replacing query-time example selection with a small set of learnable templates. The approach is conceptually simple, broadly applicable, and demonstrates consistent empirical gains across multiple datasets and two ICL backbones, with added efficiency benefits. However, the evaluation does not fully support some of the stronger claims (e.g., robustness to domain shift, two orders-of-magnitude latency gains), and comparisons to more recent and competitive ICL-based segmentation methods are missing. Additionally, the training-time overhead is not quantified, leaving the overall efficiency trade-off unclear. Despite these limitations, the core idea is sound and potentially impactful, warranting weak acceptance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors addressed my concern about optimization cost and trade-off, and showed that this method works better than Tyche-IS. I believe that this paper should be accepted.



Review #3

  • Please describe the contribution of the paper

    This paper proposes Learnable In-Context Templates, a framework for in-context medical image segmentation that replaces instance-level retrieval with optimized learnable templates. The framework consists of three stages: semantic clustering-based initialization, in-context distillation-based optimization, and multi-template inference. Evaluated on six benchmarks, LICT consistently improves Dice scores while reducing inference latency by roughly two orders of magnitude compared to retrieval-based counterparts.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper is well motivated and addresses two concrete limitations of retrieval-based ICL segmentation through a conceptually simple template-learning framework. 2.The empirical evaluation is reasonably comprehensive, including six datasets, comparisons with multiple strong baselines, efficiency analysis, and ablations on optimization design, loss terms, and cluster number, which together support the practical value of the method.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.While the method is interesting, the paper does not fully clarify its novelty relative to closely related ideas such as prototype learning, visual prompt tuning, and dataset condensation; at present, the contribution appears somewhat incremental as a combination of existing concepts adapted to ICL segmentation. 2.The analysis remains mostly performance-driven, and the paper lacks deeper interpretability or mechanistic evidence showing what the learned templates actually capture, how they differ from raw cluster-center exemplars, and why they improve robustness under domain shift. 3.The fairness of comparison could be discussed more, since the support size and context construction differ between backbones (e.g., SegGPT vs. UniverSeg), and the proposed method uses all learned templates at inference; a stricter capacity-controlled comparison would strengthen the claims. 4.The efficiency claim is convincing at inference time, but the paper does not sufficiently discuss the optimization cost of learning templates, the overall training/inference trade-off, or how the computational profile changes as the number of templates or dataset scale increases.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The method is simple and reasonably well validated, with ablations that support its design choices. However, the novelty is somewhat moderate, as the approach can be viewed as a combination of prototype-style learning and prompt optimization adapted to in-context segmentation.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I appreciate the authors’ careful and constructive rebuttal. The response addresses my main concerns reasonably well. In particular, the authors clarify the novelty of the proposed learnable in-context templates relative to dataset condensation, prototype learning, and visual prompt tuning by emphasizing the shift from raw instance retrieval to dataset-level template abstraction for ICL segmentation. They also provide a more convincing explanation of why learned templates are preferable to raw cluster-center exemplars, including an additional SNR analysis suggesting that template optimization suppresses modality-specific noise and preserves cleaner lesion morphology.

    The rebuttal also partially resolves my concerns about comparison fairness and efficiency. The authors clarify that they follow the official support-size settings for SegGPT and UniverSeg, and they provide additional evidence that the gains persist under smaller support sizes. Moreover, they quantify the offline optimization cost of template learning and explain how this cost can be amortized by the substantial reduction in inference-time retrieval overhead.



Author Feedback

We sincerely thank all reviewers for the constructive comments and address concerns below.

1.Optimization Cost and Trade-off (R2, R3) While template learning incurs an offline cost (e.g., about 50ms/image/epoch; ~50mins for TNUI’s 1K training set over 50 epochs), it eliminates repeated support-set loading and runtime retrieval, saving ~1000ms per inference. This cost is quickly amortized over TNUI’s test set (~300 cases) and becomes even more advantageous under common repeated-evaluation protocols (e.g., >10 runs). It also reduces storage I/O, benefiting clinical deployment. We will explicitly detail this training-inference trade-off in the revision.

2.Key Differences and Novelty (R1, R3) We respectfully clarify the key differences between our method and related approaches. 1) Dataset Condensation: Trains independent models but remains tied to noisy instance-level samples. 2) Prototype Learning: Extracts latent abstractions, losing the fine-grained spatial information preserved in our pixel-level templates. 3) Visual Prompt Tuning: Adapts parameters but lacks the interpretable pathological semantics captured by our morphology-preserving templates. None of the above distill dataset-level lesion geometry into explicit contextual priors for ICL segmentation.

In essence, we shift the focus from what to retrieve to what to represent—replacing noisy, redundant exemplars with compact, semantically grounded templates. These distinctions establish our method as a genuine paradigm shift, not an incremental adaptation.

➤ R1 1.DICE not optimal We thank the reviewer for the observation. While absolute Dice scores vary across datasets, our method delivers consistent, substantial gains across all six benchmarks. For example, on TNUI, it improves SegGPT [22] from 31.76% to 80.34% and outperforms Dual-SC [9] (68.93% → 80.34%). These results firmly validate our empirical effectiveness.

2.Formatting and Reproducibility We will revise the formatting issues. To ensure reproducibility, we will release all code and weights upon acceptance.

➤ R2 1.Comparison with recent baselines We sincerely thank the reviewer for these valuable references. Iris [1] unfortunately lacks public code, preventing a direct comparison. Since ICL-SAM [3] builds on UniverSeg—already included in our evaluation—we instead provide comparion with Tyche-IS [2]. Our method improves Dice from 72.81% to 86.26% on TNUI and from 53.93% to 73.06% on CVC-ColonDB, demonstrating effective transfer to newer ICL methods.

2.Domain Shift & Latency We apologize for the lack of clarity. By “domain shift,” we referred to ICL’s inherent vulnerability under zero-shot settings, not explicit cross-dataset adaptation. Our primary goal is to demonstrate broad adaptability across diverse modalities (e.g., ultrasound, colonoscopy). We will revise the manuscript to replace “domain shift” with more precise phrasing emphasizing multi-modality adaptability. We also agree the latency claim was overstated and will revise it to “a substantial order-of-magnitude reduction”.

➤ R3 1.Why learned templates are preferable We thank the reviewer for the question. Unlike raw cluster centers that retain artifacts, our templates capture distilled lesion geometry. In-context distillation suppresses modality-specific noise, raising SNR from 18.16 dB (raw training set) to 20.35 dB (learned templates). This denoising provides cleaner priors, forcing the model to focus on intrinsic morphology rather than spurious variations, directly enhancing robustness under domain shift. We will include this SNR analysis in the revision.

2.Capacity-Controlled Comparison To ensure a fair comparison, we followed the official baseline settings (K=4 for SegGPT, K=64 for UniverSeg), where our method with identical template capacity consistently outperforms. More importantly, even with non-optimal support set size, our gains persist (e.g., on SegGPT/ISIC, K=1 improves from 58.64% to 71.23% and K=2 from 61.38% to 72.74%).




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The rebuttal should clarify how the learnable templates differ from related work, and the novelty/superiority against recent medical ICL segmentation methods. Also, the authors should demonstrate the claimed domain-shift robustness and “two orders-of-magnitude” latency improvement. In addition, what the learned templates capture and why they outperform raw cluster-center exemplars need to be explained.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents a well-motivated approach to in-context medical image segmentation by replacing query-time support retrieval with compact learnable templates. Although the novelty is moderate and some claims on domain shift and latency were initially overstated, the rebuttal clarified the method’s distinction from related work. Overall, the work is sufficiently solid and accepted by all reviewers after rebuttal.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The authors have provided highly convincing and comprehensive responses in their rebuttal, successfully resolving the key technical and methodological concerns. Specifically, the authors have clearly articulated the distinction between their learnable templates and related works. Crucially, they have provided direct empirical comparisons with the recent medical in-context learning method, Tyche-IS; the proposed framework outperforms Tyche-IS, boosting the Dice similarity coefficient from 72.81% to 86.26% on the TNUI dataset, and from 53.93% to 73.06% on the CVC-ColonDB dataset, thereby solidly validating its superior adaptability and efficacy. Furthermore, the authors have properly clarified a conceptual ambiguity regarding “domain shift” in the original text, precisely redefining it as the inherent zero-shot performance degradation in in-context learning; they successfully demonstrated the model’s broad adaptability across multi-modality clinical data (e.g., ultrasound and colonoscopy) rather than explicit cross-dataset adaptation. The authors have also provided a thorough explanation of what information the learned templates capture over original cluster centers.

    By successfully decoupling clinical efficacy from high computational overhead, the proposed learnable template framework offers a novel solution that achieves both multi-modality generalization and clinical-grade inference speed for medical in-context learning.

    Consequentially, one of the reviewers raised their rating from “Weak Accept” to “Accept” following the post-rebuttal discussion. Given the strong consensus and the paper’s solid technical contribution, the AC recommends the acceptance of this manuscript.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents a clean, practical idea: replacing runtime support-set retrieval with a small set of learnable in-context templates, which improves both segmentation accuracy and inference efficiency. The method is well-validated across six datasets and two ICL backbones (SegGPT, UniverSeg), with solid ablations. Reviewer #2 turned to Accept after rebuttal, Reviewer #3 gave Weak Accept, and Reviewer #1’s Weak Reject (confidence 2) does not outweigh the overall positive consensus.



back to top