Abstract

Medical Visual Question Answering (Med-VQA) aims to provide clinical decision support by jointly reasoning over medical images and text. While existing research has made significant strides in addressing data scarcity through advanced representation learning and ensemble-based debiasing, current models still suffer from two critical issues: imbalance bias arising from skewed training distributions and modality shortcut bias where models rely on superficial linguistic patterns. In this paper, we propose Med-CAP, a counterfactual-and-calibration framework for robust Med-VQA. Specifically, to mitigate modality shortcuts, we introduce Counterfactual Evidence Modeling (CEM), which extracts prediction-relevant visual features via gradient attribution and enforces decision-making to be visually grounded through sufficiency-and-necessity constraints. To alleviate imbalance bias, we propose Adaptive Prior Suppression (APS), which calibrates fused logits by dynamically subtracting sample-adaptive language priors based on evidence strength. Experiments on SLAKE, VQA-RAD, SLAKE-CP and VQA-RAD-CP benchmarks show that Med-CAP achieves state-of-the-art robustness and accuracy across both standard and out-of-domain (OOD) settings. The code is available at \url{https://github.com/ZaiqiangHuang/Med-CAP}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3867_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/ZaiqiangHuang/Med-CAP

Link to the Dataset(s)

N/A

BibTex

@InProceedings{HuaZai_MedCAP_MICCAI2026,
        author = { Huang, Zaiqiang AND Zhu, Zhihong AND Huang, Zixuan AND Zhang, Yunyan AND Zhang, Hui AND Wu, Xian},
        title = { { Med-CAP: Counterfactual Evidence and Adaptive Prior Suppression for Robust Medical Visual Question Answering } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16878},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    1.Propose the Med-CAP framework: an anti-factual and calibration framework for robust medical visual question answering (Med-VQA), addressing the problem of medical language bias from the perspectives of imbalance bias and modal shortcuts. 2.Introduce counterfactual evidence modeling (CEM): by providing counterfactual supervision to enhance the sufficiency and necessity of visual evidence, forcing the model to make decisions based on causal-related visual evidence and avoiding reliance on shallow language patterns. 3.Propose adaptive prior suppression (APS): based on prior strength and evidence strength, adaptively suppress the overly confident only-question prior, calibrate the model’s prediction results, and alleviate the bias caused by imbalanced datasets. 4.Verify the effectiveness of the framework: conduct a large number of experiments on the SLAKE, VQA-RAD standard benchmarks, and SLAKE-CP, VQA-RAD-CP bias-sensitive benchmarks to confirm that Med-CAP not only improves the robustness against language bias but also maintains competitive intra-domain prediction accuracy.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The de-biasing effect is remarkable and comprehensive: By adopting the dual-module design of CEM and APS, it simultaneously addresses the core imbalances of Med-VQA, such as the imbalance bias and modality shortcut bias. Compared with single de-biasing methods, the robustness is significantly enhanced, especially in the OOD scenarios that are sensitive to bias. 2.Balances de-biasing and accuracy: Unlike some de-biasing methods that “sacrifice standard accuracy for de-biasing”, Med-CAP maintains the SOTA-level intra-distribution accuracy on standard datasets while enhancing the model’s anti-bias ability. It is more practical. 3.Strong module versatility and easy deployment: Based on the classic UpDn backbone network design, there is no need for large-scale model structure reconstruction. It can flexibly adapt to different Med-VQA models and has good migration and implementation capabilities.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Counterfactual supervision increases training costs: The CEM module requires constructing two types of counterfactual inputs, “evidence retention” and “evidence discard”, and calculating sufficiency and necessity constraints. Compared to traditional models, the training process is more complex. 2.Limited adaptability: The current results have only been verified on the SLAKE, VQA-RAD and their CP variant datasets. It has not been applied to more diverse types of medical images or more complex clinical question-answering scenarios. Its generalization ability needs further verification. 3.Not adapted to large-scale medical VLM: The current model is based on the traditional UpDn backbone network and has not been extended to a large-scale medical visual language model. It is difficult to fully utilize the semantic understanding ability of large models, and there is room for further performance improvement.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses the critical challenge of medical language bias in Med-VQA, proposing the Med-CAP framework with CEM and APS modules. The method is technically sound, and extensive experiments on standard (SLAKE/VQA-RAD) and bias-sensitive (SLAKE-CP/VQA-RAD-CP) benchmarks validate its effectiveness in improving robustness while maintaining in-distribution accuracy. However, several limitations prevent a higher score: 1) The framework builds on existing debiasing paradigms, with incremental rather than paradigm-shifting novelty; 2) Limited evaluation on modern medical VLMs, restricting real-world deployment impact; 3) Missing computational complexity analysis; 4) Minor ambiguities in method implementation. Overall, the work is marginally above the acceptance threshold, with final decision dependent on the authors’ rebuttal to address the above limitations.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors have provided detailed textual clarifications and additional performance numbers to directly address my concerns regarding training efficiency, dataset generalization, and modern VLM backbones. Their empirical feedback demonstrates the potential and versatility of the plug-and-play Med-CAP framework. Given their responsiveness and the commitment to integrating these implementation details into the final version, I would like to upgrade my final recommendation to a Weak Accept.



Review #2

  • Please describe the contribution of the paper

    Med CAP is a solid contribution to robust Medical VQA (Med VQA), tackling two key issues practioners deal with regularly: modality shortcut bias and dataset imbalances. The authors introduce a pipeline combining gradient attribution and counterfactual assesment.The paper builds the pipeline upon a UpDn backbone. It achieves state of the art robustness on benchmark data sets like SLAKE CP (79.97% All) and VQA RAD CP (92.24% All) .

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    I find that the Counterfactual Evidence Modeling (CEM) is a good appraoch that promotes visual grounding.Through gradient attribution, the methodenforces sufficiency (keeps evidence from the mutual domain with high GroundTursth confidence) and necessity by dropping evidence with low confidence, via a margin loss. This is a clever causal-inspired extension of prior counterfactual VQA work, and ablations show it boosts Open question accuracy meaningfully(e.g., +3.5% on SLAKE-CP). Adaptive Prior Suppression (APS) calibrates fused logits by subtracting question-only priors and scales priors confidence and evidence strength from CEM. This instance-adaptive debiasing outperforms fixed-subtraction methods like RUBi, especially on OOD biased splits, without hurting closed-ended performance too much - or that i can tell. Overall strengths: I think the dual mechanisms complement each other. CEM provides evidence quantification, and APS uses it for calibration. Strong empirical results across baselines (classical VQA, medical debiasers, VLMs), with qualitative cases showing better attention on relevant regions. The expressed design makes it practical and should be not too difficult to replicate.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1)The CEM builds directly on gradient attribution (e.g., Grad-CAM) and counterfactual VQA (e.g., Niu et al. 2021), while APS extends RUBi-style priors with adaptivity. The fusion is effective but not revolutionary, It seems like the effort is mostly hyperparameter-tuned scaling. 2) Backbone choice: UpDn (2018) is outdated; results beat recent VLMs (e.g., Qwen2.5-VL at 70.86% on SLAKE-CP), but these may be suboptimal prompts/fine-tuning. No comparison to fine-tuned modern medical VLMs (e.g., LLaVA-Med++) limits claims of broad superiority. 3)Gradient attribution limitations: The attribution seems to rely upon gradient alonfor evidence maskng, which is noisyand may miss broader context or suffer from saturation, particularly a concern with ‘real’ clinical datasets. Top-K selection with ϵ-suppression is heuristic-heavy and there are no robustness analysis to attribution failures. Granted, hard to squeeze that into a short-form paper. 4)Evaluation scope: The method works well on SLAKE/VQA-RAD(-CP), but medical VQA has more datasets (e.g., PathVQA, MIMIC-CXR). OOD is distribution-shift via priors, not true clinical shifts (e.g., new pathologies/modalities). THe authors show hyperparameter sensitivity, but use outside of benhcmark sets may not/probably wont’ perform as well.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Clearer impleentation guidance and point to code (perhaps I missed it?)

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    MedCAP makes a convincing case for improving robustness in biased MedVQA by combining counterfactual reasoning with adaptive debiasing in a well integrated way. The ablation studies clearly support its claims, and it does a good job advancing practical debiasing without overclaiming causality. The main weaknesses are that the innovation feels somewhat incremental and the backbone is a bit dated, but neither of these are dealbreakers. It would be worth checking the code to verify reproducibility.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper proposes Med‑CAP, a framework for improving the robustness of Medical Visual Question Answering (Med‑VQA) models by explicitly addressing two common failure modes: modality shortcut bias, where predictions rely excessively on question patterns rather than image evidence, and answer prior / data imbalance bias, where frequent answers dominate model behavior. The method consists of two main components. Counterfactual Evidence Modeling (CEM) encourages visual grounding by constructing evidence‑preserving and evidence‑suppressing representations and enforcing a margin‑based sufficiency/necessity objective, such that predictions are sensitive to the presence of relevant visual evidence. Adaptive Prior Suppression (APS) further calibrates the prediction logits using a question‑only prior, with a sample‑dependent suppression strength determined by both the confidence of the prior and an estimate of visual evidence strength derived from CEM. Overall, the main contribution is a unified, constraint‑based debiasing framework that integrates counterfactual visual grounding with adaptive prior correction, and demonstrates improved robustness on both standard and bias‑sensitive Med‑VQA benchmarks.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    A key strength of the paper is that it explicitly addresses two complementary sources of bias in Medical Visual Question Answering (Med‑VQA): modality shortcut bias and answer prior / data imbalance bias. Treating these failure modes jointly, rather than in isolation, is a practically meaningful design choice given how they often interact in Med‑VQA settings. The proposed framework is technically well motivated and internally coherent. Counterfactual Evidence Modeling (CEM) encourages stronger visual grounding by enforcing sensitivity to evidence‑preserving and evidence‑suppressing representations, while Adaptive Prior Suppression (APS) provides a sample‑adaptive mechanism for reducing the influence of question‑only priors. The coupling between these components via an estimated evidence‑strength signal is a reasonable and interesting way to integrate counterfactual reasoning with prior correction. The paper is also well aligned with the goals of robust and trustworthy medical AI, emphasizing failure modes under bias‑sensitive and out‑of‑distribution settings rather than focusing solely on in‑distribution accuracy. This framing is appropriate for the medical domain, where reliance on spurious correlations can be particularly problematic. Finally, the experimental evaluation is carefully designed. The use of both standard Med‑VQA benchmarks and bias‑sensitive variants allows for a more meaningful assessment of robustness, and the ablation studies help clarify the contributions of the main components of the proposed framework.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The novelty relative to recent Med‑VQA debiasing work is somewhat unclear. Related methods such as DeBCF, CCIS‑MVQA, DeCoCT, CEDO, and Med‑BiasX have already explored counterfactual or causal approaches to reducing language bias in closely related settings, so the paper would benefit from a clearer articulation of what is fundamentally new beyond the specific integration of components. In addition, the empirical gains are relatively modest and reported without variability measures (e.g., standard deviations or confidence intervals), making it difficult to assess statistical robustness. More direct head‑to‑head comparisons with the most relevant recent baselines would further strengthen the empirical claims.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I lean weak accept for this paper. The work addresses an important and practically relevant problem in Medical Visual Question Answering, namely the tendency of models to rely on superficial question patterns and skewed answer priors rather than image evidence. The proposed framework is technically sound and well motivated, and the evaluation on both standard and bias‑sensitive benchmarks suggests improved robustness in the studied setting. However, my enthusiasm is moderated by several concerns. First, the novelty relative to recent Med‑VQA debiasing work is somewhat limited, as prior methods such as DeBCF, CCIS‑MVQA, DeCoCT, CEDO, and Med‑BiasX have already explored counterfactual or causal approaches to reducing language bias in closely related settings. Second, the reported performance gains are relatively modest and are presented without variability measures (e.g., standard deviations or confidence intervals), making it difficult to assess statistical robustness. I would also have liked more direct head‑to‑head comparisons against the most relevant recent baselines and a clearer disentanglement of the contributions of the proposed components. Overall, I find the paper to be a meaningful and well‑executed methodological contribution, but not strong enough to clearly exceed the acceptance threshold without addressing these issues in the rebuttal.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank all reviewers for their constructive feedback. Below, Q refers to the questions in Section 7, and J refers to the justifications in Section 14. Response to Reviewer #1. Q1/J3: Training cost.

A: The training time is only 205 seconds on the VQA-RAD dataset with a single A100 40GB GPU. In addition, since Med-CAP introduces no additional inference-time parameters, its extra cost is training-only, and the inference cost remains the same as the original model.

Q2: Limited adaptability.

A: Following your valuable suggestion, we added more experiments on PathVQA and VQA-Med 2019.Med-CAP improves absolute accuracy by 2.28 points on PathVQA and 2.12 points on VQA-Med 2019, showing robustness on more Med-VQA benchmarks.

Q3/J2: Backbone.

A: Following your great advice, we added experiments with MedCLIP. As a plug-and-play debiasing module, Med-CAP improves the MedCLIP-based baseline by 2.56 absolute accuracy points on SLAKE and 2.74 points on VQA-RAD, showing that it is not tied to UpDn.

J1: Novelty.

A: Med-CAP has two main novelties: 1) Med-CAP effectively addresses both visual and textual biases. CEM encourages the model to base its predictions on key visual evidence, while APS adaptively reduces question-induced language priors; 2) Med-CAP can be easily incorporated into existing models without bringing additional inference-time parameters, deployment effort, or inference cost.

J4: Implementation details.

A: We will add more details, including experimental settings and key hyperparameters, in the next revision.

Response to Reviewer #2. Q1/J1: Novelty.

A: Med-CAP has two main novelties: 1) Med-CAP effectively addresses both visual and textual biases. CEM encourages the model to base its predictions on key visual evidence, while APS adaptively reduces question-induced language priors; 2) Med-CAP can be easily incorporated into existing models without bringing additional inference-time parameters, deployment effort, or inference cost.

Q2/J2: Backbone.

A: Following your great advice, in addition to UpDn, we added experiments with MedCLIP. Med-CAP still improves absolute accuracy by 2.56 points on SLAKE and 2.74 points on VQA-RAD, demonstrating the stability and generality of Med-CAP.

Q3: Attribution robustness.

A: We agree that gradient attribution may be affected by the evidence selection threshold. To verify this, we conducted a sensitivity analysis by varying top-rho in {0.1, 0.2, 0.3, 0.4} on SLAKE. The results are stable, with accuracies of 83.51, 84.26, 84.07, and 83.32, respectively.

Q4: Evaluation scope.

A: Following your valuable suggestion, we added more experiments on PathVQA and VQA-Med 2019.Med-CAP improves absolute accuracy by 2.28 points on PathVQA and 2.12 points on VQA-Med 2019, showing strong robustness.

J3: Reproducibility.

A: We will definitely release the code upon acceptance.

Response to Reviewer #3. Q1/J1: Novelty.

A: Med-CAP has two main novelties: 1) Med-CAP effectively addresses both visual and textual biases. CEM encourages the model to base its predictions on key visual evidence, while APS adaptively reduces question-induced language priors; 2) Med-CAP can be easily incorporated into existing models without bringing additional inference-time parameters, deployment effort, or inference cost.

Q2/J2: Variability. A: We ran three seeds and report the mean ± std with 95% CIs: 84.26 ± 0.20 on SLAKE (CI ± 0.50), 79.97 ± 0.18 on SLAKE-CP (CI ± 0.45), 69.18 ± 0.35 on VQA-RAD (CI ± 0.87), and 92.24 ± 0.25 on VQA-RAD-CP (CI ± 0.62), indicating stable performance.

Q3/J3: Baselines. A: Following your great advice, we conducted further experiments on PathVQA and VQA-Med 2019.Med-CAP consistently outperforms adapted CEDO by 1.08/0.87 points and Med-BiasX by 1.29/0.93 points in absolute accuracy, showing strong robustness.

J4: Components. A: CEM helps the model rely more on visual evidence, while APS reduces bias from the question text.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Several crucial concerns have been raised by all the reviewers about the limited adaptability, limited novelty, limited evaluation, unclear description. The authors are invited to submit a rebuttal to carefully respond and clarify all the concerns raised by the reviewers.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    This paper was initially invited to rebut several concerns. After carefully evaluating the authors’ rebuttal, the reviewers’ revised feedback, and re-examining the paper, I conclude that the authors have effectively addressed the key concerns raised, and all three reviewers gave positive ratings. I think the work now meets MICCAI’s publication standards.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper addresses an important problem in Medical Visual Question Answering, that is, the tendency of models to rely on language shortcuts and answer priors rather than genuine visual evidence. The main merit of the work is the unified treatment of these two complementary sources of bias through the proposed Med-CAP framework. The combination of Counterfactual Evidence Modeling and Adaptive Prior Suppression is technically sound and well-motivated. This is also aligned with the broader goal of developing robust and trustworthy medical AI systems. The experimental results consistently demonstrate improved robustness on both standard and bias-sensitive Med-VQA benchmarks. Although some limitations remain including limited evaluation on modern large-scale medical VLMs, I find that the practical importance of the problem and the consistently positive empirical results outweigh these concerns. Overall, I believe the paper provides sufficient value to the MICCAI community.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The motivation is real, and the combination of counterfactual evidence modeling and adaptive prior suppression is sensible, and ablations show that both modules contribute to standard and CP benchmarks. I would give a pass to this work.



back to top