Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

With the rapid adoption of generative AI, synthetic medical images pose growing risks, including diagnostic deception and insurance fraud. Although prior work has explored vision–language model (VLM)–based synthetic image detection, these evaluations typically consider images in isolation. In clinical practice, however, images are interpreted alongside structured records and metadata, and VLMs are increasingly deployed under joint image–record inputs. We uncover a previously underexamined multimodal vulnerability: when given both modalities, VLMs may overweight record context in authenticity judgments, such that the same image receives different predictions solely due to changes in its accompanying text. This raises concerns about robustness in real-world deployment. To systematically characterize this effect, we reformulate synthetic medical image detection as an audit of multimodal robustness at the image–record interface and introduce a paired benchmark that holds the image fixed while swapping controlled metadata variants. Across multiple imaging modalities, we evaluate diverse open-weight and frontier API VLMs and quantify how metadata alone shifts authenticity predictions. Our benchmark provides a standardized tool for assessing and improving multimodal robustness beyond image-only settings. The code is available at https://github.com/chiuhaohao/Beyond-Visual-Forensics.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2865_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/chiuhaohao/Beyond-Visual-Forensics

Link to the Dataset(s)

N/A

BibTex

@InProceedings{ChiChi_Beyond_MICCAI2026,
        author = { Chiu, Ching-Hao AND Chung, Hao-Wei AND Xu, Gelei AND Li, Xueyang AND Chen, Pin-Yu AND Kheir, John AND Ghaffari, Meysam AND Morato, Carlos AND Abbasi, Ahmed AND Shi, Yiyu},
        title = { { Beyond Visual Forensics: Auditing Multimodal Robustness for Synthetic Medical Image Detection } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16896},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper introduces a novel, controllable paired benchmark to audit the multimodal robustness of Vision-Language Models in synthetic medical image detection. The authors uncover and quantify “Text-Induced Decision Shift”, a phenomenon where VLMs overweight accompanying structured metadata over visual evidence, leading to flipped authenticity judgments on the exact same image. The study demonstrates this vulnerability across multiple datasets and frontier models.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Shifting the focus from traditional “image-only” deepfake detection to a multimodal setting (image + metadata) is highly original and much better aligned with real-world clinical workflows where images are accompanied by patient records. 2.The use of paired counterfactual metadata (holding the image fixed while changing a single provenance field) is a very effective and controlled way to isolate the impact of text on visual reasoning. 3.The introduction of the “FINAL ANSWER” vs. “VISUAL VERDICT” dual-output design cleverly disentangles whether the model’s visual perception was skewed by the text, or if the integrated decision simply yielded to the text despite correct visual perception.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.As the authors acknowledge, the text intervention is exceptionally strong. While useful as a stress test, it may not reflect the more subtle, natural variations of provenance cues or metadata inconsistencies encountered in actual clinical settings. 2.The paper does an excellent job of exposing a critical vulnerability but stops short of proposing or evaluating concrete algorithmic solutions (e. g. , specific alignment tuning, cross-attention regularization) to mitigate this multimodal bias. 3.The “I-Only” accuracy for fake images is generally low across models, meaning the models inherently struggle to detect the generated artifacts. This makes the flip analysis on fake images less robust, as the pool of initially correct predictions is small.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    The authors mention the use of a “private in-house pediatric chest X-ray dataset (PediCXR)”. However, there is no explicit statement regarding whether appropriate Institutional Review Board (IRB) approval or patient consent was obtained for utilizing, modifying, and synthesizing variations of this pediatric clinical data.

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a timely and critical blind spot in the deployment of multimodal medical AI: the vulnerability of VLMs to textual context overrides in authenticity detection. The counterfactual benchmark design is scientifically sound, and the finding that even frontier models like GPT-5 suffer from massive “text-induced decision shifts” is impactful for the MICCAI community. While the paper lacks a proposed technical fix, the exposure and thorough quantification of this deployment risk is a significant enough contribution on its own.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper addresses an important and timely issue in medical AI: how Vision-Language Models (VLMs) handle the authenticity of medical images when paired with different metadata. The authors introduce the concept of “Text-Induced Decision Shift,” showing that the same image can be judged as real or fake depending on the accompanying provenance text. The study uses a clever “image-fixed, metadata-swapped” benchmark across several datasets (NIH-CXR14, ISIC2019, PediCXR) and model families. This is a well-motivated setup that highlights a potential “shortcut” or over-reliance on text cues in multimodal medical forensics

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Effective Experimental Design: The paired evaluation is the strongest part of the paper. By keeping the image identical and only changing the metadata, the authors clearly isolate the effect of text on the model’s final decision.
    • Significant Empirical Findings: The reported drops in accuracy are quite eye-opening. For instance, seeing MedGemma-27B’s performance on real images fall from 97.6% to 0.0% when labeled as “Source-AI” is a very strong way to demonstrate the model’s vulnerability.
    • Comprehensive Evaluation: Testing across three different medical datasets and a wide variety of models (from medical VLMs to general-purpose frontier models) makes the findings much more convincing.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Explicitness of Metadata Cues: The use of “Source: AI-edited” is a very strong and direct cue. It is somewhat expected that a language-driven model would shift its prediction when given such an explicit label. While this works well for a stress test, it might not fully capture the more subtle or natural ways metadata is recorded in actual clinical workflows.
    • Details on the Generation Pipeline: The synthetic image generation process would benefit from more transparency. Since the benchmark relies on images generated by Gemini, it would be helpful to know the failure rates, how clinician agreement was handled during validation, and how the authors addressed potential biases in the sampling strategy.
    • Independence of ‘VISUAL VERDICT’: The paper uses a “VISUAL VERDICT” step to analyze the model’s reasoning. However, because this verdict is generated in the same pass as the “FINAL ANSWER” after the model has seen the text metadata, it might not be a completely independent visual reference. The metadata likely already influenced the “visual” reasoning part of the output.
    • Reporting of Statistical Uncertainty: The results would be more robust if they included fuller numerical tables and some form of uncertainty estimates (like confidence intervals) to show that these shifts are consistent across different samples.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • Discussion on Natural Metadata: I suggest adding a discussion on how this decision shift might manifest with more “natural” or less explicit clinical notes, rather than just the direct “Source-AI” tag.
    • Methodological Transparency: Please clarify the clinician review process, specifically how many experts were involved and how you handled cases where they disagreed on the quality of a synthetic image.
    • Visual vs. Multimodal Baseline: To better understand the “VISUAL VERDICT” entanglement, it would be helpful if the authors could discuss (or provide a small sample of) how the models perform on the images alone, without any metadata at all.
    • Reporting Robustness: Including confidence intervals or a brief look at how sensitive the models are to different prompt phrasings would make the empirical results even stronger.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper tackles a very interesting and relevant problem—how text can bias the visual judgment of VLMs in medical forensics. The “image-swapping” idea is elegant, and the results are quite striking. However, the current manuscript leaves some methodological questions open, particularly regarding the synthetic image validation and the very explicit nature of the metadata intervention. If the authors can provide more details on the generation pipeline and discuss the practical clinical implications of these shifts during the rebuttal, I would be happy to reconsider my score.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I thank the authors for their detailed rebuttal. Their clarification regarding the synthetic image generation pipeline, failure rates, and clinician involvement adequately addresses my concerns about methodological transparency. In addition, I appreciate the authors’ candor in explicitly framing the ‘Source: AI-edited’ intervention as an upper-bound stress test; this effectively establishes a clear vulnerability in VLMs. However, the lack of evaluation on more subtle, natural clinical metadata shifts remains a limitation for real-world applicability. Nevertheless, the transparency of the rebuttal and the foundational value of this stress test, combined with their thorough response addressing my main methodological concerns, justify my decision to update my independent assessment from Weak Reject to Weak Accept.



Review #3

  • Please describe the contribution of the paper

    This article presents a standard of auditing multimodal robustness in synthetic medical image detection. The main concept is to retain the image constant and modify only the metadata that accompanies it so that the study can quantify the ability of text alone to alter an authenticity decision of a vision-language model. The article demonstrates that metadata can have a significant effect on many VLMs despite the visual evidence remaining the same, which is an essential deployment risk in the context of multimodal medical processes in the real world.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The article examines a significant and relevant issue, i.e. the possibility of multimodal medical AI systems being misinformed by metadata when determining whether an image is real or synthetic. 2.The benchmark design is straightforward and intuitive: the identical picture is combined with manipulated metadata variants, and one can directly isolate the text-induced changes in decisions. 3.The paper is highly motivated in terms of deployment, particularly to clinical and insurance-related workflows where image authenticity can be a concern. 4.The analysis includes several datasets and several model families such as open-weight medical VLMs, general VLMs, frontier API models, and synthetic-image detection VLMs, which enhances the applicability of the results. 5.The primary empirical result is practical and helpful: metadata per se can substantially alter authenticity judgments, implying that image-only judgments can be biased in underestimating multimodal risk. 6.The article presents a practical auditing instrument that may aid future studies on strong multimodal authenticity evaluation.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The paper primarily offers a benchmark and audit framework as opposed to a specific approach to enhancing robustness, so its contribution is more diagnostic than solution-focused. 2.The provenance cue used in the metadata is deliberately strong, making the benchmark a useful upper-bound stress test, but also raises the question of how well the results would be representative of more natural real-world metadata variations. 3.Reproducibility is limited. Although the general architecture is outlined, a few details about the implementation would have been more explicit to allow full replication, particularly regarding data generation, prompting, and evaluation processes. 4.Part of the reported gains or declines are possibly due to poor baseline performance on fake-image detection, so some flip behaviors are more difficult to interpret. 5.The paper would be more robust with a larger range of metadata perturbations not just one source/provenance field and more discussion of the extent to which those metadata perturbations are realistic in clinical practice.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I lean weak accept because the paper identifies a meaningful and underexplored vulnerability in multimodal medical AI, namely that metadata can steer authenticity judgments even when the image remains unchanged. The benchmark is conceptually clear, the evaluation spans multiple datasets and model families, and the results are practically relevant. My reservations are that the benchmark is intentionally stress-test oriented, the work is more diagnostic than solution-focused, and some reproducibility details remain limited. Even so, I believe the paper makes a useful contribution as an auditing benchmark for multimodal robustness.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I have read the authors’ rebuttal carefully. The rebuttal addresses my main concerns in a reasonable way. In particular, the authors clarify that the Source: AI-edited metadata cue is intended as a standardized upper-bound stress test rather than a direct replication of all real-world clinical metadata variation. This clarification helps frame the contribution more appropriately as a controlled audit of multimodal robustness.

    The rebuttal also provides useful additional details about the synthetic image generation and validation pipeline, including failed edit attempts, clinician review, bootstrap standard errors, and plans to release prompts, code, and the benchmark upon acceptance. I also appreciate the clarification that VISUAL VERDICT is not intended to be a fully independent image-only judgment, but rather a same-pass consistency probe, while the main text-induced decision shift claim relies on paired FINAL ANSWER comparisons.

    Some limitations remain. The metadata intervention is still deliberately strong, and the paper would benefit from future work evaluating more natural and subtle metadata perturbations. The work is also primarily diagnostic rather than solution-focused. However, I believe the paper makes a timely and useful contribution by exposing an important multimodal robustness risk in synthetic medical image detection. Therefore, I maintain a positive recommendation and support acceptance.



Author Feedback

Q1: Intervention Design, Metadata Realism and Benchmark Scope (R1, R2, R3, Meta) A: This work is informed by our collaborating domain experts from a leading health insurance company, where AI-assisted documentation is entering clinical workflows and provenance information is becoming visible to downstream systems: on authentic images, an AI-origin cue can trigger unnecessary fraud investigations; on synthetic images, hospital-like provenance can suppress skepticism and enable improper payouts. Yet this issue remains underexplored.

Because these cues are highly heterogeneous and difficult to quantify consistently across healthcare environments, we adopted a standardized field as a reproducible upper-bound stress test, not a replication of real-world metadata diversity. During the design phase, we explored alternative wordings and observed varying degrees of shift, suggesting the vulnerability persists beyond the specific cue used. We chose the stronger yet uniform intervention to clearly expose the failure mode. Systematically characterizing cue strength and extending to multi-field perturbations are important but distinct challenges, as each introduces its own confounders. Our benchmark serves as a first-step controlled audit, and we hope it draws community attention to multimodal robustness risks at the image-record interface and stimulates work on broader perturbations and mitigation strategies.

Q2: Generation Pipeline, Clinician Validation, Reproducibility and Statistical Reporting (R2, R3, Meta) A: As described in Sec. 2.1, each image is edited via an LLM-guided edit-verify-refine loop. Failed edits receive structured feedback on the violated criterion and are re-executed on the original image for up to five rounds. The pipeline discards on average 1.2 (NIH-CXR14), 0.1 (ISIC2019), and 0.4 (PediCXR) failed attempts per retained image. Two clinicians reviewed CXR and dermoscopy, respectively, excluding implausible cases through conservative quality control. Although the retained set may exhibit selection bias because some edits are easier to generate, this does not undermine our core claim, since the analysis performs paired comparisons on the same image. For prompting, metadata is placed after the task description under multimodal conditions. Bootstrap resampling (1000 iterations) on accuracy yields standard errors of 1.58% (NIH-CXR14), 1.53% (ISIC2019), and 2.47% (PediCXR), indicating low sampling variability. The full prompt, benchmark, and code will be released upon acceptance.

Q3: Independence and Role of VISUAL VERDICT (R2, Meta) A: We agree VISUAL VERDICT is not a fully independent image-only judgment, since it is generated after the model has seen metadata. The true independent baseline is I-Only (Fig. 2(A)), addressing R2’s request. VISUAL VERDICT is a same-pass consistency probe distinguishing two failure modes: (1) FINAL ANSWER shifts while VISUAL VERDICT does not, and (2) metadata shifts both together. I-Only vs. I+Source cannot reveal this, as they involve separate inference passes. This entanglement means our split analysis is conservative: VISUAL VERDICT may be shifted by metadata, making outputs appear consistent and reducing the observed Verdict Split Rate. Importantly, our claim on Text-Induced Decision Shift relies on paired FINAL ANSWER comparisons, not on VISUAL VERDICT.

Q4: Low I-Only Baseline on Fake Images (R1, R3) A: Low I-Only accuracy on fake images limits the flip analysis pool. However, combined with I+Base and I+Source-AI, this is an important finding: current VLMs struggle to detect realistic synthetic medical images visually, and Base Metadata does not reliably help. As in Sec. 3.2, gains only under Source-AI suggest reliance on the provenance cue rather than improved visual capability. This is the vulnerability our benchmark aims to capture.

Q5: IRB Approval for Private Dataset (R1) A: PediCXR was used under institutional IRB approval; details were omitted for anonymity.



Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper introduces a focused benchmark for auditing multimodal robustness in synthetic medical image detection by holding the image fixed while changing only a provenance field in the accompanying metadata. The reviewers generally agree that the problem is timely and important, and that the paired counterfactual design is a clean way to isolate text-induced shifts in VLM authenticity judgments. The evaluation across multiple datasets and model families further supports the relevance of the finding. The main concerns are about the strength and realism of the intervention, rather than the basic validity of the benchmark. In particular, the Source: AI-edited cue is very explicit and may represent an upper-bound stress test rather than typical clinical metadata variation. The authors should clarify this framing and, if possible, discuss or evaluate more natural metadata perturbations. The rebuttal should also provide more detail on the synthetic image generation and clinician validation process, including failure rates, number of reviewers, disagreement handling, and any potential bias from using Gemini-generated edits. Finally, the VISUAL VERDICT should be interpreted carefully, since it is generated in the same pass after exposure to metadata and may not be an independent image-only judgment. Additional reproducibility details and prompt.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    I recommend acceptance. The rebuttal addressed the Meta-AC and reviewers’ main concerns by explicitly framing the “Source: AI-edited” cue as a standardized upper-bound stress test rather than a model of natural clinical metadata, detailing the edit-verify-refine generation pipeline with per-dataset failure rates and two-clinician validation, reporting bootstrap standard errors of 1.58%, 1.53%, and 2.47%, clarifying that VISUAL VERDICT is a same-pass consistency probe while the core Text-Induced Decision Shift claim relies on paired FINAL ANSWER comparisons, and confirming IRB approval for PediCXR. These clarifications led R2 to raise the score from Weak Reject to Accept, while R1 and R3 maintained their positive assessments. The lack of evaluation of more subtle metadata perturbations and the absence of a proposed mitigation remain limitations, but they are design-level issues that fall outside what can be addressed in rebuttal under MICCAI policy. The controlled audit contribution is timely and useful for the community.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper is accepted. The reviewers agree that it addresses an important and timely deployment risk in multimodal medical AI: VLM vulnerability to textual context overrides in authenticity detection. Although the work is best framed as an upper-bound stress test rather than a full simulation of natural clinical metadata shifts, the benchmark is scientifically useful, the rebuttal addressed key transparency concerns, and the findings are likely to stimulate meaningful discussion in the MICCAI community.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents an audit of vision-language models for detecting synthetic/fake images when these are accompanied by different metadata. In the first round of reviews, reviewers raised concerns about: i) limited implementation details, ii) lack of evidence against real-world text interventions, and iii) potential low baseline task performance. In the rebuttal, the authors answered these concerns and convinced R2 and R3.

    In my opinion, the paper should be accepted for the following reasons:

    • Clear motivation, timely and relevant challenge ;
    • Exhaustive evaluation across 3 datasets and a wide variety of models (from medical VLMs to general-purpose frontier models) ;
    • Elegant, while marginally novel, auditing method. While R3 remains skeptical about the use of strong-text interventions as an upper-bound stress test, I believe this paper highlights a timely, important issue that truly needs such a stress test to raise awareness in the community. As proposed by the reviewers, I suggest that the authors add the implementation details to the camera-ready version and emphasize the “upper-bound stress test” characteristic of this audit.



back to top