Abstract

Medical vision-language models (VLMs) enable zero-shot clin- ical image classification, yet reliably detecting out-of-distribution (OOD) inputs at deployment remains an open problem. No static scoring method works across all shift types: Maximum Concept Matching (MCM) on FLAIR achieves 76.4% AUROC for far-OOD but only 42.4% for covariate shifts such as ultra-wide-field fundus images —effectively random. We trace this to a structural mismatch: covariate-shifted inputs are indistinguishable from in-distribution samples in softmax space, yet occupy distinct regions in the VLM’s embedding space. To exploit this untappe signal, we propose PROTON (PROtotype-based Test-time ONline OOD detection), a lightweight post-hoc module that maintains an online prototype bank from high-confidence test predictions and adaptively fuses prototype distance with MCM scoring via stream-level variance statistics, requiring no model modification, training data, or prompt engineering. On the ophthalmology benchmark (FLAIR + FIVES), PROTON improves MCM by +23.9 AUROC on covariate shift, +8.8 on semantic, and +8.1 on far-OOD —the only zero-shot method to improve all three without hierarchical prompts or labeled data.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4773_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{DasAbh_PROTON_MICCAI2026,
        author = { Das, Abhijit AND Wasalathilaka, Nichula AND Lu, Yifan AND Dukre, Adinath AND Mahapatra, Dwarikanath AND Khan, Shadab AND Razzak, Imran},
        title = { { PROTON: Prototype-Based Test-Time Online OOD Detection for Medical VLMs } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The work identifies that existing OOD detection methods in probability space (e. g. , MCM ), using contrastive VLMs as the backbone, mainly target semantic shift. However, in the medical domain, detecting covariate shift is equally important. To address this, it incorporates feature-space information via an online-updated memory bank.

    The prototype bank is initially constructed using confidence scores from the softmax outputs. After processing multiple data streams, class prototypes are formed by averaging class embeddings, and the distance to these prototypes is used as the OOD score. The final OOD score is a weighted combination of the MCM score and the proposed embedding-based score.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Strengths:

    • The paper is well-written and the identified problem (\i. e. , detecting both semantic-shift and covariate-shift) is interesting to the community.
    • Extensive experiments have been done to valid the effectiveness of the proposed framework.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Weaknesses:

    • It is true the the proposed framework is free from accessing the training data, however, it necessities to access test samples. Meanwhile, the construction of the prototype bank initially relies on the reliability of confidence score (\i. e. , MCM), however, one of motivations of this work is that MCM is unreliable to detect the covariate-shift OOD samples. Imagine that if the beginning sampels are mostly covariate-shift OOD samples, then the quality of prototype bank might be degraded.
    • I believe the authors miss one important work TAG [1], which does not rely on the model modification, training data, and prompt engineering. The authors are encouraged to incorporate the results of OOD detection using TAG as one of baselines.

    [1] TAG: Text Prompt Augmentation for Zero-Shot Out-of-Distribution Detection, ECCV, 2024.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    See weaknesses.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    Thanks for conducting the suggested experiments. Q1: The authors evaluate PROTON across various ID_fraction settings, showing consistent improved performance compared to MCM. Q2: The authors add suggested baseline.



Review #2

  • Please describe the contribution of the paper

    The paper proposes PROTON, a lightweight prototype-based test-time online OOD detection framework for medical vision-language models. Its main contribution is to exploit the embedding-space structure of test samples by maintaining an online prototype bank and combining prototype distance with MCM scoring through adaptive variance-based fusion. This allows the method to improve OOD detection across covariate, semantic, and far-OOD shifts without requiring training data, prompt engineering, or model updates.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The proposed online prototype bank does not require access to training data, which is particularly valuable in medical applications where data privacy and restricted data sharing are major practical concerns. 2.The method shows clear empirical gains over strong baseline methods, with especially significant improvements on covariate shift while also consistently improving semantic and far-OOD detection. 3.The ablation and sensitivity studies suggest that the method is robust to hyperparameter choices, which strengthens its practicality for real deployment settings.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.A limitation of the paper is that the method is mainly validated on top of MCM. The authors claim that PROTON is compatible with other scoring functions, but they do not provide enough empirical evidence to show that it can reliably improve non-MCM scores as well. 2.The empirical comparison is not sufficiently broad. In particular, only a small number of non-MCM-style baselines are considered, and these are relatively old methods, which weakens the strength of the experimental validation. 3.The method strongly depends on high classification accuracy of the base model. If the model makes incorrect predictions, the prototype bank can be contaminated and may even fail completely. However, the paper does not discuss this sensitivity or analyze performance when the classifier is weak.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My recommendation is based on limited confidence in the paper’s broader impact. The proposed method appears to be mainly an improvement over MCM, so it is unclear whether the idea has more general value beyond that specific setting. In addition, the set of baselines is relatively limited, which makes it harder to judge the true strength of the method against a broader range of alternatives. Finally, the experimental analysis is not thorough enough, and several important questions about robustness and applicability remain insufficiently explored.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposes PROTON which is a lightweight post-hoc module for out-of-distribution (OOD) detection in medical vision-language models (VLMs). The core insight is that covariate-shifted inputs (e.g., ultra-wide-field fundus images) produce high softmax confidence indistinguishable from in-distribution samples, yet are geometrically separable in the VLM’s embedding space. PROTON maintains an online prototype bank built from high-confidence test predictions via FIFO queues, computes a prototype distance score, and adaptively fuses it with MCM scoring using Welford’s online variance estimation. The method requires no model modification, training data, or prompt engineering. On the FLAIR + FIVES ophthalmology benchmark, PROTON improves MCM by +23.9 AUROC on covariate shift, +8.8 on semantic shift, and +8.1 on far-OOD.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    I think the paper identifies a clear and well-characterized failure mode. The observation that covariate-shifted inputs are indistinguishable in softmax space but separable in embedding space (Figure 1) is compelling. The t-SNE plots, MCM score overlap analysis, and 2D scatter of MCM vs. prototype distance all provide direct evidence, which I find convincing. This is not just a hand-wavy motivation.

    The method design is also elegant. PROTON adds no learnable parameters, requires a single forward pass, and uses O(1) memory per update via Welford’s algorithm. The confidence gate to prevent OOD contamination of prototypes is a small but important detail. I think the idea of using MCM variance to automatically detect when softmax confidence becomes unreliable (Eq. 3-4) is clever. The sigmoid interpolation avoids hard switching, and the behavior analysis showing αt dropping when covariate samples enter demonstrates the mechanism works as intended.

    When I look at the ablation (Table 2), I appreciate that it systematically decomposes contributions: prototype distance (+15.5 covariate AUROC), adaptive fusion (+4.8), and confidence gate (+3.6). The γ × M sensitivity analysis across three medical domains (Figure 3b) also shows the method is robust across a wide range of hyperparameters, which is reassuring for practical deployment.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    My main concern is that the full evaluation is conducted on a single benchmark (FLAIR + FIVES) for the main results. The ID set contains only 300 samples (150 Normal + 150 DR) and OOD sets are similarly small. For a method that relies on online prototype accumulation, the stream length matters. The γ × M heatmaps in Figure 3b show results on radiology and pathology, but only as ∆AUROC without absolute numbers or dataset details. I would have liked to see full Table 1-style results on at least one additional domain.

    Also, the method assumes the test stream contains a sufficient proportion of in-distribution samples to build meaningful prototypes. In deployment scenarios where most incoming samples are OOD (e.g., a screening system receiving many misrouted images), the prototype bank could be contaminated despite the confidence gate. The paper does not analyze sensitivity to the ID/OOD ratio in the stream, and I am curious about how the method degrades in such scenarios.

    Another point I think is that hierarchical MCM (L=5) achieves 87.7% covariate AUROC versus PROTON’s 66.3%, which is a gap of over 20 points. While the authors note this requires domain-expert prompt trees, the performance difference is substantial and I think should be discussed more openly. The prototype convergence analysis (Figure 3a) is also shown only for radiology and pathology, not for the primary ophthalmology benchmark, which is an odd choice.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    The paper is clearly structured. I find the “shift-type triage” idea mentioned in the Discussion (using the two-score decomposition to distinguish covariate vs. semantic vs. far-OOD) interesting but underdeveloped. Even a simple proof-of-concept would strengthen the paper. The σ²₀ reference threshold (set to 0.02) is described as “nominal ID-only variance” but how this value is chosen and whether it is sensitive to VLM architecture or dataset should be clarified.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I think the idea is new and fresh. Using online prototypes in the VLM’s embedding space to catch covariate shifts that softmax-based methods completely miss is a well-motivated contribution. The method is simple, parameter-free, and immediately deployable, which I find practically valuable. The core insight is convincingly demonstrated and the ablations are clean. The main limitations are the single-benchmark evaluation and the notable gap with hierarchical MCM, but these do not undermine the core contribution. I lean toward weak accept since the idea is sound and the direction is promising for the medical imaging community.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    After reading the rebuttal, I think my questions were mostly addressed. I would remain my score as weak accept. Thanks. Also, please make this work reproducible regarding the code and the model.



Author Feedback

We thank reviewers for recognizing PROTON’s novelty, motivation, and clean ablations. We address all the comments below.

[R2.1] Beyond MCM. PROTON needs only a scalar base score S_b and embedding e_t: S_PROTON^b = a_tS_b_hat + (1-a_t)S_proto_hat; MCM is one instantiation. Across six base scores, PROTON improves every one on all shifts (mean +11-15 AUROC). Cov AUROC (base->+PROTON): MaxLogit 42.8->62.5; GL-MCM 43.5->65.8; Energy 43.8->62.0; MCM 42.4->66.3; Entropy 44.1->63.1; TAG-MCM 45.7->67.1.Semantic and far-OOD follow the same pattern. The contribution is image-embedding geometry — a complement to any softmax/logit score, not an MCM trick. [R2.2] Broader baselines. We add TAG-MCM (ECCV’24), Hier.MCM(L=5), and privileged ID-feature banks. Cov AUROC: TAG-MCM 45.7; kNN-ID 70.8, Mahalanobis-ID 68.6 (labeled ID features); Hier.MCM(L=5) 87.7 (LLM prompts); PROTON 66.3 (no train, no prompts); Hier.MCM+PROTON 88.4.PROTON matches privileged feature-bank methods without training data or prompts, and composes with Hier.MCM. [R2.3] Sensitivity to classifier accuracy. Under forced misrouting, PROTON stays above MCM through 30% misrouting (65.5 vs 57.6 mean). The confidence gate (g=0.7) alone adds +3.6 cov-AUROC (Tab.2) by blocking OOD contamination; pure-MCM fallback (Alg.1) holds before K_min. We will add a conservative gate (confidence+margin+prototype-consistency) with base-score fallback when unstable.

[R1.1] Cold-start and online setting. PROTON is strictly online: x_t is scored using the bank from past accepted samples; the bank updates only after (B_t = Update(B_{t-1}, e_t) iff max_c p_{t,c} >= g). No labels, transduction, or model updates. Before all classes reach K_min, PROTON falls back to pure MCM (Alg. 1). The score is a fusion S_PROTON = a_tS_MCM + (1-a_t)S_proto — MCM only gates which samples enter the bank; the OOD decision uses S_proto, capturing the embedding separation MCM misses. Stream ID-fraction (Sem/Cov/Far AUROC): rho=90%: 63.2/67.4/84.8; rho=70%: 62.7/66.3/84.5 (default) rho=30%: 58.8/58.5/80.4; rho=10%: 55.1/50.9/77.2; MCM: 53.9/42.4/76.4 PROTON stays above MCM down to 10% ID. Stream-order CIs (5 perms): 62.7+-1.1 / 66.3+-1.8 / 84.5+-0.9 (order-stable). The non-zero ID-fraction assumption will be stated explicitly. [R1.2] TAG baseline. Added — see [R2.2]. TAG-MCM (45.7 cov-AUROC) underperforms PROTON (66.3) and is orthogonal: TAG augments text prompts, PROTON exploits image-embedding geometry. Using TAG-MCM as PROTON’s base score yields 67.1. [R3.1] Cross-domain / cross-VLM. We replace dAUROC heatmaps with absolute Table 1-style numbers across three VLMs/domains. Cov AUROC (MCM->PROTON): Ophth. (FLAIR) 42.4->66.3; Radio. (UniMedCLIP) 44.7->90.4; Patho. (QuiltNet) 45.3->84.2.Gains are largest on covariate shift; semantic and far-OOD also improve everywhere (full metrics in appendix). Datasets: Radio. = Normal/Pneumonia (RSNA, 300 ea.); Patho. = Benign/Malignant (PCam-derived, 300 ea.). [R3.2] ID/OOD stream-ratio. See [R1.1]- PROTON stays above MCM down to 10% ID. In low-ID screening, the confidence gate+K_min fallback keep prototypes uncontaminated; pure-MCM is recovered in the worst case, so PROTON never degrades below baseline. [R3.3] Hier.MCM gap and FLAIR convergence. The 21-point cov-AUROC gap is real: Hier.MCM uses LLM-generated prompt trees, PROTON uses none. They are complementary — Hier.MCM+PROTON reaches 88.4 cov-AUROC, and PROTON exceeds Hier.MCM on far-OOD (84.5 vs 82.4). PROTON’s distinct value: zero training data, prompt trees, or expert input. The FLAIR convergence plot will be added (cos to final prototype >0.99 within ~55 samples; parallels Fig.3a). [R3.4] sigma_0^2 and shift-type triage. sigma_0^2 is not tuned per-VLM: sweeping {0.005-0.08} gives flat mean AUROC 69.8-71.2 (sigmoid saturation). 0.02 corresponds to early-stream ID variance from a brief warm-up. The (S_MCM, S_proto) decomposition supports shift-type triage with macro-F1 = 0.70 on 3-way Cov/Sem/Far classification.




Meta-Review

Meta-review #1

  • Your recommendation

    ActiveX VT_ERROR:

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    N/A

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    N/A

  • Please justify your recommendation.

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    This paper presents a well-motivated and elegant, parameter-free framework for online test-time OOD detection in medical vision-language models, effectively addressing both semantic and covariate shifts. In the post-rebuttal phase, the authors comprehensively resolved the core technical critiques by demonstrating robust performance across six distinct base scoring functions and multiple clinical modalities, earning the clear support of the majority of the panel. Although one reviewer maintained a negative recommendation regarding broader impact, the strong empirical gains, thorough validation, and consensus among the other experts lean firmly toward acceptance.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents PROTON, a framework for OOD detection with contrastive VLMs that addresses an important limitation of existing works where they focus on semantic shift while ignoring covariate shift, which is highly relevant in medical settings. By incorporating feature-space information through an online-updated memory bank, the method offers a clinically meaningful and technically useful extension. During the review process, reviewers identified an important baseline comparison, which the authors incorporated in the rebuttal. The updated results further demonstrate the strength of the PROTON framework, and reviewers also acknowledged these improvements, shifting toward acceptance. Given the method’s contribution, clinical relevance, and comprehensive analysis, this work represents a solid contribution to MICCAI, and I recommend acceptance.



Meta-review #4

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Two reviewers recommend acceptance, and R2 recommends rejection. R2 points out that the proposed method has limited impact beyond the MCM method. R2 also points out limited baselines. However, MCM is usually indeed a strong baseline. Also, the number of comparative methods is limited in the online scenario without training data. Finally, R3 points out the limited experimental evaluations; however, the authors explore various medical VLMs and modalities, which, in my opinion, suffice for a conference version. Based on the above, despite their validity, I do not believe the remaining concerns invalidate the work, which is an interesting, practical contribution to OOD detection using VLMs. Therefore, I recommend acceptance.

    Note: The FLAIR reference appears to be incorrect. It points out to a natural image VLM, not a medical VLM. Please correct this issue.



back to top