Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Medical foundation models have achieved remarkable clinical performance, yet their robustness under real-world perturbations remains underexplored. We present a robustness benchmark comprising 40 perturbation types (12 base, 28 medical-specific) across eight imaging modalities, evaluating five VLMs (LLaVA-Med, MedGemma, MedGemma-1.5, Gemini-2.5-flash and GPT-4o-mini) on VQA, visual grounding, and captioning, alongside two segmentation models (MedSAM, SAM-Med2D) with five fine-tuning strategies. Our findings reveal: (1) Fine-tuning strategy dominates robustness, with LoRA exhibiting nearly double the degradation of full fine-tuning, while SAM-Med2D’s Adapter offers favorable efficiency-robustness trade-off. (2) Medical-specific perturbations disproportionately damage segmentation, with 9 of 15 top corruptions being domain-specific. (3) LoRA-tuned visual grounding drops over 40 points, whereas zero-shot captioning remains stable (<7\% drop). Zero-shot VQA shows model-dependent robustness—medical models drop under 20\% while Gemini-2.5-flash drops 54\%. General-purpose VLMs achieve higher VQA accuracy but fail on grounding; among medical VLMs, MedGemma demonstrates the best overall stability. These results provide deployment guidelines and underscore the necessity of domain-specific robustness evaluation for medical AI. Our code is available at: https://abnerai.github.io/MedFM-Robust.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2490_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/AbnerAI/MedFM-Robust

Link to the Dataset(s)

N/A

BibTex

@InProceedings{CuiXia_MedFMRobust_MICCAI2026,
        author = { Cui, Xiangxiang AND Huang, Tianjin AND Wang, Yifang AND Hu, Lijie AND Yin, Lu},
        title = { { MedFM-Robust: Benchmarking Robustness of Medical Foundation Models } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16895},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper
    • A modality-adaptive perturbation suite with 12 generic corruptions and modality-specific clinical-style artifacts (40 types total across eight imaging modalities), calibrated into five SSIM-guided severity bands so that degradation is comparable across modalities and perturbations.
    • A unified robustness benchmark that evaluates medical VLMs (VQA, captioning, visual grounding) and SAM-style segmentation (MedSAM, SAM-Med2D) under a shared perturbation philosophy, bridging dense prediction and vision–language understanding.
    • Systematic evidence on how adaptation strategy shapes robustness (e. g. , full fine-tuning vs LoRA vs adapter), including task-asymmetric protocols (zero-shot VQA/captioning vs LoRA-tuned grounding) and deployment-oriented takeaways (e. g. , efficiency–robustness trade-offs).
  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The perturbation design is meaningfully closer to deployment than generic natural-image corruptions (ImageNet-C [3]): it explicitly models acquisition-linked artifacts (e. g. , CT streaking/cupping, MRI ghosting/bias field, ultrasound shadowing/reverberation, pathology stain shifts, endoscopy specular highlights/bubbles, OCT shadow/blink/defocus, X-ray scatter/exposure/grid effects). The paper’s point that domain-specific failures can dominate top failure modes is substantiated (e. g. , many of the hardest corruptions are medical-specific), which is a strong “data contribution” for a benchmark paper.
    • SSIM-guided severity calibration with binary search per corruption type is a practical way to align “how broken” the image is across modalities, improving fairness of cross-modality robustness comparisons where pixel-level noise scaling is not commensurate.
    • Reusing established public benchmarks for VLMs (OmniMedVQA [5], ROCOv2 [15], MeCoVQA [6]) focuses effort on evaluation protocol extension rather than introducing yet another proprietary dataset, which aids comparability and adoption.
    • Segmentation experiments span heterogeneous clinical datasets (e. g. , ISIC 2016 [2], Kvasir-SEG [8], brain MRI tumor, glaucoma disc/cup), enabling dataset-sensitivity discussion (e. g. , endoscopy vs MRI stability) that is valuable for benchmark interpretation.
    • Metrics and robustness reporting are straightforward and interpretable: task-standard scores (IoU/Dice; VQA accuracy; grounding IoU@0.5; BLEU/ROUGE-L/CIDEr) plus absolute clean-to-perturbed drops (Δ), with analyses across severity, perturbation families, and fine-tuning strategies.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • VLM evaluation uses only 500 samples per task; statistical stability for rare concepts/modalities may be limited. The authors mention fixed random sampling (e. g. , seed 42 as indicated in the released code); without stratification or sensitivity analysis across multiple seeds/splits, representativeness of conclusions for long-tail clinical cases remains a question.
    • Synthetic corruptions still differ from real multi-site PACS variability, mixed artifacts, and reconstruction pipelines; external validation on real degraded cohorts would strengthen claims about real clinical world reliability, beyond plausible simulation.
    • Modality coverage in messaging vs experiments: eight modalities are emphasized for perturbation design, but segmentation coverage is concentrated in five benchmark datasets; not all 40 corruptions apply uniformly to every VLM benchmark modality. A compact matrix (dataset → applied med-specific corruptions) in the appendix would improve reproducibility and reduce reader confusion when figures show illustrative organs/modalities not always present in the evaluated split.
    • Protocol asymmetry: VQA/captioning are zero-shot while grounding is LoRA-adapted; this is understandable but makes cross-task robustness comparisons not “like-for-like. ” The limitation should be stated prominently to avoid over-interpreting grounding robustness as purely “VLM robustness” rather than “LoRA localization robustness under corruption. ”
    • API VLMs (Gemini, GPT-4o-mini) complicate exact replication; even with released code, if preprocessing/model snapshots are not fully pinned, leaderboard-style numbers may drift. A quick look at the anonymous repository suggests inputs may be resized to 224 in places, but model versioning/decoding policies may still be under-documented for full third-party reproduction.

    • Caption robustness is summarized heavily via n-gram metrics; clinically, factual consistency/hallucination risk may matter more than BLEU stability. This is a common benchmark limitation but worth stating explicitly for medical AI researchers.

    • These limitations are not fundamental threats to the contribution, and the benchmark’s scope, clinical relevance of the corruptions, and unified evaluation narrative substantially outweigh the concerns noted above.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
    • This study scales across modalities/datasets and proposes a unified evaluation framework spanning heterogeneous tasks (VLMs and segmentation), which is valuable for the MICCAI conference.
    • Domain-specific corruption modeling is thoughtfully developed, and the released code makes the protocol largely reproducible, supporting follow-on robustness research in medical foundation models.
  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The modality-adaptive pipeline innovatively blends 12 base and 28 medical-specific corruptions, which enables fair cross-task comparisons which is missing in prior clean-data evaluations. Fine-tuning strategy rankings (full FT > adapters > LoRA) are convincingly visualized and tabulated per dataset, guiding practical choices like SAM-Med2D adapters for efficiency.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Main contribution of this article is the broad coverage of 8 modalities and 5 datasets which strengthens generalizability, with clear evidence that medical corruptions drive 9/15 top failures.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Dataset size varies across all modality. Small datasize raises concerns regarding the statistical significance of the results.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Comments:

    (1) How do authors validate synthetically generated perturbations in CT and MRI are clinically accurate? (2) How robust are medical foundation models (including both vision-language models and segmentation models) to real-world, domain-specific perturbations across multiple medical imaging modalities and tasks, and how do different fine-tuning strategies impact this robustness? (3) Are 500 samples sufficient to prove the statistical significance of the results e.g. drop in 2x drop of LoRA’s robustness (4) Authors must care to expand few necessary abbreviations.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The modality-adaptive pipeline innovatively blends 12 base and 28 medical-specific corruptions, which enables fair cross-task comparisons which is missing in prior clean-data evaluations. Fine-tuning strategy rankings (full FT > adapters > LoRA) are convincingly visualized and tabulated per dataset, guiding practical choices like SAM-Med2D adapters for efficiency. Main contribution of this article is the broad coverage of 8 modalities and 5 datasets which strengthens generalizability, with clear evidence that medical corruptions drive 9/15 top failures.

    Major concern: Dataset size varies across all modality. Small datasize raises concerns regarding the statistical significance of the results. Comments:

    (1) How do authors validate synthetically generated perturbations in CT and MRI are clinically accurate? (2) How robust are medical foundation models (including both vision-language models and segmentation models) to real-world, domain-specific perturbations across multiple medical imaging modalities and tasks, and how do different fine-tuning strategies impact this robustness? (3) Are 500 samples sufficient to prove the statistical significance of the results e.g. drop in 2x drop of LoRA’s robustness (4) Authors must care to expand few necessary abbreviations.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The primary contribution of this paper is the comprehensive benchmark designed to evaluate the robustness of Medical Foundation Models (MedFMs) against real-world clinical perturbations. The framework evaluates five Vision-Language Models (VLMs) and two SAM-based segmentation models across eight medical imaging modalities. To achieve this, the authors engineered 40 distinct perturbation types (12 base/generic and 28 modality-specific) calibrated into five severity levels using the Structural Similarity Index (SSIM). Furthermore, the paper systematically investigates how five different fine-tuning strategies (including full fine-tuning and LoRA) impact a model’s resilience to these perturbations.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The authors meticulously designed 28 modality-specific perturbations that simulate actual clinical artifacts, such as CT metal artifacts, ultrasound acoustic shadowing, and MRI k-space motion. This approach is highly validated by their finding that 9 out of the top 15 most fatal corruptions to models are domain-specific, proving that generic noise is insufficient for evaluating medical AI.

    2.The paper provides a critical, highly practical insight for clinical AI deployment in aspect of the choice of fine-tuning strategy profoundly dictates a model’s robustness. Specifically, the authors convincingly demonstrate that Parameter-Efficient Fine-Tuning (PEFT) methods, particularly LoRA, exhibit nearly double the performance degradation under noise compared to full fine-tuning. This serves as a vital warning for practitioners who might use LoRA for its computational efficiency without considering its safety trade-offs in noisy clinical environments.

    3.The benchmark is highly commendable for its breadth, unifying the evaluation of both dense prediction tasks (segmentation) and complex vision-language understanding tasks (VQA, Visual Grounding, and Captioning) under a single, cohesive framework.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.From a clinical perspective, SSIM is a somewhat improper proxy for medical severity. For instance, a highly localized artifact that completely obscures a small, malignant tumor might result in a very high SSIM (as the global image barely changes) but is clinically fatal. Conversely, widespread benign noise (like mild speckle) might drastically lower the SSIM but still allow for a perfect diagnosis. Calibrating noise purely on global pixel statistics ignores the localized, diagnostic significance of medical image features.

    2.The authors evaluated VQA and Captioning in a zero-shot setting, whereas Visual Grounding was evaluated using LoRA fine-tuning. The paper later concludes that Visual Grounding suffers a catastrophic robustness drop compared to the stable zero-shot tasks. However, because the authors already proved that LoRA itself is highly vulnerable to noise, it is impossible to disentangle the cause of this failure. Is Visual Grounding inherently sensitive to noise, or is the catastrophic drop simply an artifact of using the fragile LoRA technique?

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper makes a highly valuable and necessary contribution to the medical AI community by exposing the fragility of foundation models under clinical noise.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the Area Chair and all reviewers for their positive assessment and for recognizing the benchmark’s clinical relevance and deployment value. We address the main concerns below. Q1.Statistical significance of 500 samples (R1, R2). We argue that 500 samples is sufficient for our conclusions for two reasons. First, the effect sizes we report far exceed any noise from sample variance. LoRA shows nearly 2× the degradation of full fine-tuning (0.048 vs 0.025 IoU), and LoRA-tuned grounding drops over 40 points. These are not marginal differences. Second, the trends are consistent across independent datasets, models, and perturbation types, providing convergent evidence beyond any single split. Beyond sufficiency, we note that the combinatorial scale of our evaluation (5 VLMs × 3 tasks × 40 perturbations × 5 severity levels, totaling hundreds of thousands of inferences including paid API calls) makes substantially larger sample sizes computationally and financially prohibitive for community-scale reproducibility. Q2.Protocol asymmetry: zero-shot VQA/Captioning vs LoRA-tuned grounding (R1, R3). We thank R3 for raising this. The choice of protocol is driven by feasibility, not design preference. Zero-shot performance is acceptable for VQA and Captioning, allowing meaningful robustness evaluation, while zero-shot grounding is essentially non-functional (general-purpose VLMs score 0–0.1%, Fig. 3(h)), making LoRA adaptation the only viable evaluation setting. Importantly, the three tasks are evaluated as independent studies under their respective realistic deployment protocols, rather than as a head-to-head cross-task comparison. We do not claim grounding is “inherently more fragile” than VQA/captioning. Rather, our finding is that under the protocols in which each task is actually usable, LoRA-adapted grounding shows substantially larger robustness degradation than zero-shot VQA/captioning, which is itself a deployment-relevant observation. We will clarify this framing in §3.3 and reframe the claim as “LoRA-adapted grounding robustness” in the camera-ready. Q3.SSIM as a severity proxy (R3). SSIM is used as a modality-agnostic engineering anchor for cross-modality degradation comparability, not as a clinical severity metric. Without such an anchor, perturbation magnitudes are not commensurate across modalities (e.g., a noise level meaningful for CT may be trivial for ultrasound). We will clarify this scoping in §2.1.Q4.Reproducibility (R1, R2). Our anonymized repository (linked in the abstract) will be further organized and open-sourced to support replication. Beyond the points above, we will further refine the manuscript in the camera-ready version to incorporate the reviewers’ suggestions wherever feasible. We thank the reviewers again for feedback that helps sharpen the paper’s framing.



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper is accepted because it provides a timely and practically valuable benchmark with broad modality and task coverage, spanning 28 clinically motivated perturbations across medical imaging settings and both dense prediction and vision-language tasks.

    The reviewers particularly appreciated that the perturbation design goes beyond generic natural-image corruptions and is supported by clear evidence that medical-specific artifacts constitute many of the most severe failure modes.

    The work also offers actionable insights for clinical deployment, especially regarding severity calibration, robustness reporting, and the safety trade-offs of fine-tuning strategies such as LoRA versus full fine-tuning.



back to top