List of Papers Browse by Subject Areas Author List
Abstract
Medical vision-language models (Med-VLMs) have demonstrated strong performance on medical visual question answering, yet they remain prone to hallucination, generating clinically unsupported statements that are insufficiently grounded in image evidence. Mitigation methods applied during decoding offer a practical solution, but they typically lack anatomical awareness or rely heavily on ground truth annotations, which limits their applicability. We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a framework that operates entirely during inference and requires no manual annotations for anatomically grounded hallucination mitigation. CAST automatically discovers anatomical regions relevant to the given query through broad medical segmentation. It then selects a compact, causally informative area using counterfactual intervention based on the drop in answer likelihood under occlusion. Guided by this chosen region, CAST performs a unified contrastive decoding process, combining classifier-free guidance to correct spatial attention with stepwise temporal contrast to regulate generation dynamics. Experiments on the SLAKE and MIMIC-CXR datasets across three Med-VLMs demonstrate that CAST consistently outperforms strong baselines and surpasses decoding strategies reliant on ground truth. Our results indicate that compact, automatically selected regions provide highly effective contrastive guidance without expert annotations, offering a practical and generalizable solution for improving spatial grounding and reducing hallucinations.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4832_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/csyifan/CAST
Link to the Dataset(s)
MIMIC-CXR dataset: https://physionet.org/content/mimic-cxr/
SLAKE dataset: https://huggingface.co/datasets/BoKelvin/SLAKE
BibTex
@InProceedings{LuYif_Counterfactual_MICCAI2026,
author = { Lu, Yifan AND Dukre, Adinath AND Das, Abhijit AND Zou, Ziyun AND Yang, Haolin AND Xie, Yutong AND Razzak, Imran},
title = { { Counterfactual Anatomy-Guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), an inference-time framework designed to mitigate hallucinations in Medical Vision-Language Models (Med-VLMs). It introduces an annotation-free region proposal stage utilizing MedSAM3 to generate multiple anatomical candidate masks. It presents a counterfactual region selection method that identifies the most causally relevant region by measuring the drop in the model’s predictive log-likelihood when the region is occluded using blur and mean operators. It applies a dual spatial-temporal contrastive decoding strategy, using classifier-free guidance (CFG) for spatial attention and a step-wise temporal contrast to regulate token generation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Valid Methodological Insight: The authors correctly observe that ground-truth (GT) bounding boxes in medical datasets are often too coarse, and suppressing them removes too much visual context, weakening the contrastive signal during decoding. The push for compact masks is well-motivated. 2.Impressive Closed-ended Performance: The method shows substantial improvements in factual grounding, notably increasing closed-ended accuracy on the SLAKE dataset from 61.42 to 81.49 for Phi-3.5V-Med. 3.Plug-and-Play Design: Operating entirely at inference time without requiring model retraining is highly desirable for deploying large medical foundation models.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Ignored Computational Catastrophe: The paper severely glosses over the inference latency. For every single question, the system must: 1) Run MedSAM3; 2) Generate a draft answer via greedy decoding; 3) Perform two occlusion forward passes (blur and mean) for up to $K=16$ candidate regions to calculate the causal effect. This means running the VLM over 33 times just to select a region before the final decoding even begins. This computational overhead makes the method completely impractical for real-world clinical use, yet the authors provide zero latency/FLOPs analysis. 2.Misleading “Annotation-Free” Claim: The framework is not truly zero-shot or annotation-free; it merely outsources the annotation dependency to MedSAM3.If MedSAM3 has poor zero-shot performance on a specific sub-modality (e. g. , a rare ultrasound view), the candidate proposals will fail, causing the entire pipeline to collapse. 3.Heavy Reliance on Heuristics and Hyperparameters: The method introduces an excessive number of hyperparameters: mask area penalty ($\lambda$), variance penalty ($\mu$), guidance scale ($\gamma$), inside/outside ROI weights ($\alpha$, $\beta$), and temporal weights ($w_m$, $w_p$). Relying on empirical “grid search” to tune these limits the claimed “plug-and-play” generalization.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The fundamental idea of using counterfactuals to find compact anatomical regions for contrastive decoding is mathematically sound and empirically effective for closed-ended queries. However, the paper is completely oblivious to its own computational cost. Running a VLM 30+ times per query for region selection defeats the purpose of an “efficient” inference-time intervention. Furthermore, the reliance on an upstream segmenter (MedSAM3) shifts the dataset bias problem rather than solving it. For this to be accepted, the authors must provide a rigorous inference-time analysis (latency/throughput) in their rebuttal and justify the stability of their numerous hyperparameters.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I have no further concerns.
Review #2
- Please describe the contribution of the paper
This paper proposes a decoding-time method that mitigates hallucination in Med-VLM’s output. It first uses SAM for region proposal, then selects question-related regions, and finally decodes with spatial-temporal hallucination mitigation. This method is versatile and training-free, making it easy to apply to existing state-of-the-art systems. The results show that the overall performance is the best.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) The motivation is well-introduced. Hallucination is a common research topic and training-free make the model more generalizable. 2) The proposed method is logically sound. It smartly combines existing methods’ segmentation ability to enable annotation-free settings. 3) The experiments are persuasive. The effectiveness of the proposed method has been validated on three commonly-used medical VLM on both MIMIC and SLAKE datasets.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1) The evaluation is limited to visual question answering task, while free-text report generation should also be a crucial ability of medical VLMs. 2) The proposed method can cause significant delay on response time. It would be great if author/s can acknowledge this and report the inference time before and after adding the proposed mechanisms. 3) It lacks failure case analysis.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the proposed method is not open-sourced and not fully-validated on all critical tasks in medical VLMs, it introduces a way to generate more evidences and produce grounded output without human labor. Given the generalizability and effectiveness of the proposed method, I would recommend Accept.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Given the space limit of MICCAI and the overall soundness of the proposed idea, I consider the single-task evaluation and the reliance on external models/annotations to be acceptable limitations
Review #3
- Please describe the contribution of the paper
This paper proposes Counterfactual Anatomy-guided Spatial-Temporal Decoding (CAST), an annotation-free, inference-time framework for hallucination mitigation in medical vision-language models (VLMs). The method first generates candidate anatomical regions using a MedSAM3, then selects a compact and question-relevant region via a counterfactual likelihood drop criterion. Based on the selected region, CAST performs spatial-temporal contrastive decoding to suppress language-driven hallucinations. Experiments on the closed-ended set of SLAKE and MIMIC-CXR demonstrate improvements over existing decoding-time baselines.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Hallucination mitigation in medical VLMs is an important problem, and decoding-time approaches are attractive due to their plug-and-play nature.
2.CAST removes the dependency on ground-truth ROI annotations by leveraging automatic region discovery, which improves practical applicability.
3.The method shows strong gains in closed-ended accuracy across multiple models and datasets.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The paper claims that compact masks provide a cleaner contrastive signal than coarse masks, but does not provide direct experimental evidence to support this claim.
2.CAST relies on segmentation-based anatomical regions. However, for questions involving global properties (e.g., imaging modality or general conditions), it is unclear how the method would operate effectively.
3.The paper argues that compact regions are preferable, but this may not always hold. Preserving some surrounding context could be beneficial for diagnosis, as the contrast between lesion and background is often important.
4.The intuition that “the prior step’s distribution encodes language momentum” is not clearly justified. It remains unclear why subtracting the previous-step logits leads to better grounding, and how this mechanism interacts with spatial contrast during decoding.
5.While CAST performs strongly on closed-ended questions, its performance on open-ended questions is often inferior to existing methods (as shown in Table 1), which raises concerns about the general effectiveness of the method.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The main issues are the unclear motivation behind key design components and the consistently weaker performance on open-ended data compared to existing methods.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers. R1(well-motivated, substantial improvements), R3 (recommended Accept); R4 (improves practical applicability). We hope our responses address the concerns and justify updating the remaining ratings to Accept.
Inference latency: Wall-clock per question on Phi-3.5V-Med/SLAKE is: Baseline 3.13s, OPERA 3.15s (1.00×), VCD 3.48s (1.11×), ARCD 3.58s (1.14×), DoLA 6.64s (2.12×), CAST 6.68s (2.13×). CAST overhead matches DoLA. MedSAM3 proposals are precomputed once/image and read from disk; online MedSAM3 would add only ~0.13s/question. ROI scoring uses teacher-forced likelihood over a short draft (T_score=32), not full autoregressive generation; each occlusion pass takes ~256 ms, below a ~3s generation. K candidates can run in parallel for further gains.
Annotation-free and MedSAM3 dependence: Annotation-free means no per-dataset segmentation labels at inference, as in CLIP or SAM pipelines. CAST handles MedSAM3 failure by design: MedSAM3 only proposes candidates, while Stage 2 selects via Δℓ_mean−λ·area+μ·consistency. Semantically wrong proposals receive near-zero causal signal in both perturbation modes and cannot enter the top K=16 rankings. The area-prior fallback in Sec.3.3 covers the residual 1.3% of images with fewer than three valid proposals. CAST relies on causal discriminability, not anatomical exactness, as compact auto masks with about 7% coverage consistently outperform coarse GT boxes with about 39% coverage.
Hyperparameters: The same defaults (γ=1.5, α=0.01, β=3.0, λ=μ=0.1, wm=1.0, wp=0.5, K=16) are reused across all three Med-VLMs and both datasets, without per-cell tuning. γ follows the standard CFG range; α,β follow ARCD; wm:wp=2:1 is one fixed ratio. Fig. 2 confirms stability: removing λ lowers overall by 3.4%; single-occlusion variants drop 3.3–4.0%; spatial-only and temporal-only drop to 53.66% and 53.91% vs. full 59.25%.
Compactness/context/temporal contrast: We don‘t claim compact masks are always preferable. We claim overly large masks weaken ROI-guided CFG as suppressing them removes too much information from the unconditional branch. Fig.2 is an ablation holding the VLM and decoder fixed and varying only mask source. The mask compactness analysis (Sec.3.2) shows GT coverage >40% noticeably reduces closed accuracy, and GT slightly reduces accuracy versus NULL decoding. Sec.3.1 explains: as area(m)→1, log p_c−log p_u collapses to a global correction, eliminating localized contrastive signal. CAST keeps context: the conditional branch always sees the full image, while the unconditional branch suppresses ROI tokens (α=0.01) and upweights surrounding tokens (β=3.0), amplifying lesion-versus-background contrast. Compactness describes the contrastive signal, not the model input. Formally, ℓ_CFG(t−1) approximates prior-token logit bias; subtracting wp·ℓ_CFG(t−1) penalizes tokens already favored before the latest visual re-anchoring, implementing step-wise debiased decoding aligned with Anti-LM and contrastive decoding. Ablating wp=0 reduces SLAKE closed accuracy, confirming its role.
Open-ended analysis: The open/closed gap largely reflects SLAKE/MIMIC open-ended scoring rather than poorer decoding. It uses token recall against very short references (avg. ≈1.7 tokens), so lexical mismatch is penalized despite semantic correctness: color questions with GT Black/Gray get 0 when VLMs answer descriptively, and abnormality questions (≈18% of open queries) with rigid GTs such as Pneumonia or Lung Cancer reject synonyms like pulmonary infection. These errors stem from the base VLM’s lexical surface form and are mostly invariant to token-level contrastive decoding. CAST’s contribution is to suppress hallucinated affirmations on diagnostically loaded queries, captured by closed-ended evaluation; it yields consistent gains there (+13.7 on SLAKE) while staying neutral on open-ended. We view this as the desired safety trade-off in medical VQA, where confident false claims pose the main clinical risk.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper is well motivated in addressing hallucinations in medical VLMs and received mixed reviews. To better demonstrate its contributions, the authors should further analyze inference efficiency and include failure case studies. More details can be found in the reviewers’ comments.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The author provides detailed feedback, and the rebuttal has addressed most of the concerns raised by the reviewers. Although one reviewer hasn’t sent the feedback, the other two are happy with the rebuttal. The contribution is acceptable for MICCAI.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper addresses the important issue of hallucination mitigation in Medical Vision-Language Models (VLMs) by proposing an inference-time, annotation-free spatial-temporal decoding framework. Initially, reviewers expressed concerns regarding the computational overhead, the true “annotation-free” nature of the method, the reliance on numerous hyperparameters, and the weaker performance on open-ended questions.
The authors’ rebuttal effectively addressed these critical points. They provided a detailed latency analysis showing that the overhead matches existing baselines like DoLA and that the MedSAM3 proposals can be precomputed. They clarified how the method robustly handles poor region proposals and demonstrated the stability of their hyperparameter choices across different models and datasets without per-cell tuning. Furthermore, they adequately explained that the open-ended performance gap is largely an artifact of the rigid lexical scoring metrics rather than a flaw in the decoding strategy itself. As the rebuttal resolved the primary computational and methodological concerns—leading one skeptical reviewer to upgrade their recommendation to Accept—the paper is recommended for inclusion.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
CAST is a strong inference-time contribution. It removes the need for ground-truth ROI annotations by selecting compact counterfactual anatomy masks and then applying spatial-temporal contrastive decoding. But the method depends on a capable medical segmentation model and multiple forward/occlusion passes, so the practical latency and failure modes under poor segmentation need more attention in their future work.