Paper Info Reviews Author Feedback Meta-Review Back to top

List of Papers Browse by Subject Areas Author List

Abstract

Vision–language models (VLMs) are increasingly used in medical imaging, yet their robustness to spurious correlations remains insufficiently characterized. We propose a controlled evaluation framework that uses synthetic artifacts modelled after common acquisition confounders to parametrically vary correlation strength, and pairs two complementary test protocols — artifact removal and artifact inversion — to isolate whether models rely on clinical features or visual shortcuts. Applying the framework to diabetic retinopathy grading in fundus photography and BI-RADS-based assessment in mammography, we evaluate five architectures spanning a spectrum from no concept supervision to full multi-level image–concept alignment. We find that VLM backbones retain clinical signal when shortcuts are absent, yet actively follow spurious associations when they conflict with pathology, degrading faster than standard visual backbones — a dual encoding that is only exposed when evaluation goes beyond clean test sets. Among concept-based strategies, only architectures that both reshape the feature space toward clinical concepts and shield the classifier from non-clinical signal provide meaningful resilience. The framework is architecture-agnostic and applicable to any vision or multimodal model. To support evaluation in mammography, where the combinatorial richness of the BI-RADS lexicon cannot be feasibly captured by binary concepts alone, we release expert annotations with pixel-wise delineation of findings for 400 images. They can be found, alongside the code, at the following \href{https://github.com/valecorbetta/framework_for_vlm_eval}{link}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5420_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/valecorbetta/framework_for_vlm_eval

Link to the Dataset(s)

https://github.com/Emory-HITI/EMBED_Open_Data https://csyizhou.github.io/FGADR/ https://zenodo.org/records/20849422

BibTex

@InProceedings{CorVal_Beyond_MICCAI2026,
        author = { Corbetta, Valentina AND Portaluri, Antonio AND He, Muzhen AND Boeke, Daniël AND Beets-Tan, Regina AND Lachi, Veronica AND Wetzer, Elisabeth AND Jenssen, Robert AND Silva, Wilson AND Wickstrøm, Kristoffer},
        title = { { Beyond Clean Test Sets: Spurious Correlations in Medical Vision-language Models and the Role of Concept Supervision } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}

Reviews

Review #1

Please describe the contribution of the paper

This paper addresses the problem of spurious correlations and shortcut reliance in medical vision–language models (VLMs). It proposes a controlled evaluation framework that uses synthetic artifacts to parametrically vary the strength of spurious correlations. Two test protocols—artifact removal and artifact inversion—are introduced to assess whether models rely on genuine clinical features or visual shortcuts. The authors further study the role of concept supervision, arguing that robust behavior requires architectures that both align representations with clinical concepts and shield classifiers from non-clinical signals. Experiments compare the robustness of five architectures, ranging from no concept supervision to full multi-level image–concept alignment, on DR grading and mammography assessment tasks. The authors also plan to release a BI-RADS dataset with pixel-wise expert annotations.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper tackles an important and timely problem. Robustness to spurious correlations is critical for medical foundation models.

The proposed evaluation framework is conceptually clear and methodologically sound. By combining synthetic confounders with artifact removal and artifact inversion, the framework provides a more informative robustness analysis than standard in-distribution testing alone.

The observation that models may learn clinical features and shortcuts in parallel—performing well when the artifact is absent but following the artifact when it conflicts with pathology—is interesting .

The use of t-SNE visualizations provides a straightforward and interpretable way to illustrate model behavior under different confounding conditions.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

Limited external validity of synthetic artifacts: While synthetic artifacts enable controlled experiments, they may not reflect real-world clinical confounders, which are often more subtle and complex. This limitation restricts the external validity of the conclusions. The paper would benefit from a more explicit discussion of the gap between synthetic artifacts and real clinical confounding factors.

Inconsistent and confusing performance trends: It is unclear why MICA exhibits the worst accuracy at the initial stage despite incorporating concept supervision. From Figure 2 (top right), MICA’s performance appears to increase as the spurious correlation percentage rises, which is counterintuitive. Under artifact inversion, MICA also shows a larger performance drop (at p=100%) than some non-ViL baselines. Overall, the results are mixed and do not consistently support the claimed conclusions.

Insufficient justification of concept supervision benefits: The experimental results do not convincingly demonstrate that adding concepts (e. g. , via PCBM or MICA) consistently improves robustness. In some cases, concept-supervised models perform similarly to or sometime worse than simpler baselines.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The authors claimed to release the source code and/or dataset upon acceptance of the submission.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

N/A
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I lean toward a weak accept, as the proposed evaluation framework seems to be conceptually clear and methodologically sound to me, and it addresses a critical issue in medical vision–language models—namely, robustness to spurious correlations and shortcut learning.
Reviewer confidence

Somewhat confident (2)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

N/A
[Post rebuttal] Please justify your final decision from above.

N/A

Review #2

Please describe the contribution of the paper

This paper addresses a critical problem in medical imaging: the robustness of vision-language models (VLMs) to spurious correlations. The authors propose a controlled evaluation framework that systematically injects synthetic artifacts mimicking real acquisition confounders at varying prevalence levels. They evaluate five architectures with different levels of concept supervision across two clinical domains (diabetic retinopathy grading and mammography assessment) using two complementary test protocols: artifact removal and artifact inversion. The key finding is that VLMs encode both clinical and artifactual signals in parallel, following spurious cues when they conflict with pathology despite retaining clinical signal when shortcuts are absent.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- It tackles an important yet underexplored problem. While medical VLMs are increasingly deployed, their robustness to spurious correlations in realistic scenarios has been insufficiently characterized. The observation that existing benchmarks evaluate on data where multiple confounders co-occur, making it difficult to isolate specific effects, is particularly insightful.
- The complementary test protocols (artifact removal and artifact inversion) are interesting and necessary. Testing under artifact removal alone would not reveal the dual encoding behavior discovered in this work. The parametric control of artifact prevalence allows for systematic evaluation rather than binary present/absent comparisons.
- Annotations of BI-RADS dataset for 400 mammography images from the EMBED dataset, extending existing labels to the complete lexicon with pixel-wise delineation will be a nice addition for the research community. This addresses the combinatorial complexity of mammographic descriptors that cannot be captured by binary concepts alone.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The evaluation covers only two clinical domains and two VLM backbones (RetCLIP and Mammo-CLIP). Both domains involve 2D imaging with relatively well-defined pathologies. It remains unclear whether the findings generalize to 3D imaging (CT, MRI), multi-organ systems, or more subtle pathological patterns. The framework’s claimed “architecture-agnostic” nature is supported theoretically but not demonstrated empirically beyond the tested models.
- While the synthetic overlays are modeled after real artifacts, they are parametrically generated and may differ from true acquisition confounders in ways that affect model behavior. Real artifacts often involve complex interactions with anatomical structures (e.g., eyelash shadows varying with gaze direction, grid artifacts interacting with tissue density). The paper would benefit from validation showing that model behavior on synthetic artifacts predicts behavior on real ones, even if only on a small held-out set.
- The paper focuses primarily on methodological evaluation but provides limited guidance for practitioners. When should concept supervision be preferred over standard VLM fine-tuning? What level of spurious correlation prevalence warrants intervention? How should the choice of clinical concepts be validated? These practical questions receive minimal attention.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

N/A
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper makes meaningful contributions to an important problem and demonstrates good experimental methodology. It proposes a novel evaluation framework with complementary test protocols that reveal behaviors obscured by standard evaluation and valuable data contribution (BI-RADS annotations). However, it has some limited scopes and missing analyses such as limited evaluation scope (two domains, two VLM backbones) raises questions about generalizability, synthetic artifacts may not fully capture real-world confounder complexity and Limited practical guidance for practitioners.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

Authors clarified some of the concerns. They should include these clarifications in the final manuscript for proper clarity.

Review #3

Please describe the contribution of the paper

The authors propose a controlled, architecture-agnostic framework for evaluating domain-specific medical VLMs under injected spurious correlations with parametrically varying strength. They further provide evidence that these models encode clinical and artifactual signals in parallel, and show that substantial robustness is obtained only when the architecture both grounds the representation in clinical concepts and prevents non-clinical information from reaching the classifier.
Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1.The paper is motivated by a practical question: whether domain-specific medical VLMs that exhibit implicit clinical grounding through contrastive pre-training remain vulnerable to spurious visual correlations.

2.The authors compare five architectures ordered along an interpretable axis of concept supervision. This design helps isolate the contributions of each design choice and provides a clear diagnostic insight that can be easily followed.
Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

1.While the controlled experiments are well designed, the empirical scope remains somewhat narrow for claiming robustness across domains, especially given the limited scale of the mammography study. More fundamentally, both domains are evaluated on single-site data, yet the paper’s motivation centers on institutional confounding in VLM pre-training. Additional evaluation on larger cohorts and external-site validation would both strengthen the robustness claims and directly test the problem the paper frames.

2.The paper is well structured and the framing around Q1-Q3 gives the experiments and results a clear narrative. However, the writing in sections such as the introduction to concept-based methods are fairly dense and could be clearer.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

N/A
Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

1.The artifact-inversion protocol appears to rely on a single cyclic reassignment of artifact-class associations. Since different cyclic shifts may induce slightly different difficulty profiles, averaging results across all non-identity cyclic shifts (e.g., the three possible shifts in DR) would better rule out shift-specific effects while adding little computational overhead.

2.Section 1 has a typo in “difficult o isolate”.
Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This complementary design of artifact removal and artifact inversion in the evaluation framework is well-motivated and provides a meaningful comparative study. However, the empirical study is relatively limited in scale, and the robustness claims are demonstrated primarily under controlled synthetic confounders rather than external real-world distribution shifts. As a result, the broader claims about robustness across domains would be more convincing with larger-scale validation and additional ablations, and the current evidence leaves some uncertainty regarding the generalizability of the conclusions.
Reviewer confidence

Confident but not absolutely certain (3)
[Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

Accept
[Post rebuttal] Please justify your final decision from above.

The authors have provided reasonable context for the scale limitations and committed to discussing them explicitly in the limitations section, and they have committed to addressing the concerns regarding dense writing. While the external-validation gap remains a real limitation of the work, it is reasonably scoped given the data-annotation constraints the authors describe, and the paper’s core contributions stand.

Author Feedback

We thank the reviewers and the area chair for their feedback.

AC, R1, and R2 Synthetic artifacts: controlled setting and realism. “Clean” could be clearer: in real-world test data, any spurious correlations are difficult to control or identify. Our artifact removal/inversion protocol is the controlled analogue. To clarify: dual encoding cannot be detected from a clean (or removal) test set alone: a model following artifacts can still appear robust when they are absent. Inversion provides the counterfactual, jointly exposing dual encoding. This kind of controlled setting is what real medical data cannot provide (institutional confounders co-vary with class but cannot be set, swept, or inverted), motivating synthetic overlays, analogously to Waterbirds-style benchmarks (Sagawa et al. 2019). Our overlays are modelled after real acquisition confounders (Fig. 1), but appearance fidelity is a separate question: rigorous validation requires a reader study with clinicians on matched synthetic and real cases, sweeping the generation parameters at adequate power, beyond this work’s scope. We will note this as a limitation and future direction.

R1 Inconsistent MICA trends. Our primary robustness metric is the slope of balanced accuracy across p, not absolute accuracy at any p. R1’s three observations are consistent with the framework working as designed: (a) Lower clean accuracy at p=0% on DR. Stage 2’s concept bottleneck imposes a documented capacity cost (Koh et al. 2020). The MICA authors [2] recover accuracy by training Stage 2 on full features; in our setting this would weaken robustness, as Stage 1’s encoder grounding requires classifier shielding to be fully effective. On mammography at p=0%, MICA matches the VLM baseline (0.41), so this cost is task-dependent. (b) Increase with p on mammography removal. MICA’s DR values from p=0 to p=75% (0.53–0.55) lie within stds — no significant trend; the mammography is similarly flat with more fluctuation, due to the noisier task [11]. The slope (+0.3×10⁻³, vs. the VLM baseline’s negative slope, ∗p<0.05) is invariance, not artifact exploitation. (c) Drop under inversion at p=100%. In DR, MICA is the only model above chance (0.25 vs. ~0.20), with the smallest drop (0.28 vs. 0.36–0.43). In mammography, convergence to 0.33 at p=100% reflects the task’s difficulty [11]: under sufficient inversion, models default to uniform output. The trajectory matters more than the endpoint: MICA holds ~0.41 from p=0% through p=75%, while every other model has already collapsed to chance by p=50. R1, R2 Concept supervision and practitioner guidance. Recent work positions concepts as a route to robustness as well as interpretability [10, 3, 25]; we entered this study expecting the same. Our findings instead show two things the community has not yet emphasised: (1) domain-specific VLMs are already more robust than expected, and (2) the concept-based interventions tested provide smaller and more conditional gains than the literature suggests. We therefore guide practitioners away from assuming concepts deliver robustness “for free”: their inclusion and clinical validation is costly, and the field needs deeper investigation before recommending such pipelines clinically.

R2, R3 Scope, clarity, typo. DR and mammography differ substantially (RGB vs. grayscale, distinct organs). Datasets with concept annotations are scarce in medical imaging; DR has [31], while for mammography we provide, to our knowledge, the first dataset annotated with the complete BI-RADS lexicon and pixel-wise delineation, produced by clinicians with deep BI-RADS expertise. This bounds the mammography sample size and is a contribution we hope the community will build on. Extending to 3D imaging, multi-organ, multi-site cohorts, and additional cyclic-shift permutations would enrich the analysis; we will discuss these in §4.We will also revise §1’s concept-methods paragraph for readability and correct the typo.

Meta-Review

Meta-review #1

Your recommendation

Invite for Rebuttal
Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Overall the paper presented interesting, controlled experiments to study spurious correlation in medical concept VLMs. For such a study, experiments should strongly support the arguments. The authors are invited to address the concerns from reviewers, especially “Inconsistent and confusing performance trends” and “Insufficient justification of concept supervision benefits” as per Reviewer #1; discuss the limitations regarding the differences between synthesized and natural artifacts, the guidance for practitioners as per Reviewer #2; and discuss potential external generalization.

The AC also encourage the authors to clarify what’s a “clean test set” and why the dual encoding can only be revealed by a clean test set.
After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

The authors have clearly addressed the concerns, and all reviewers as well as me agree to accept the paper

Meta-review #2

After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

Accept
Please justify your recommendation.

This paper presents a framework to evaluate shortcut reliance in vision-language models using synthetic artifacts. In two datasets spanning two domains, the authors test artifact removal and inversion to measure robustness. The experimental measure of robustness is based on representations’ alignment with clinical concepts and classifiers’ immunity to non-clinical signals. The reviewers were quite convinced by the quality of the proposed framework, raising several minor points regarding generalizability due to the limited scale of the evaluation. However, the authors clearly addressed the reviewers’ concerns, leading R2 and R3 to revise their ratings to accept. Therefore, I recommend acceptance.

back to top

Beyond Clean Test Sets: Spurious Correlations in Medical Vision-language Models and the Role of Concept Supervision

Author(s):