List of Papers Browse by Subject Areas Author List
Abstract
While medical Multimodal Large Language Models (MLLMs) have shown promise in assisting diagnosis, they still frequently generate hallucinated responses that appear linguistically plausible but lack visual evidence. Such hallucinations pose risks to clinical decision-making and necessitate effective detection. Existing introspective detection methods primarily perform uncertainty estimation or logical verification by analyzing model responses conditioned on original or perturbed inputs. However, such external perturbations are often heuristic and context-agnostic, which overlooks the internal cross-modal dependency between generated tokens and related visual tokens during decoding. To address this issue, we propose VIHD, a Visual Intervention-based Hallucination Detection method that leverages targeted visual token masking to calibrate semantic entropy for more effective hallucination detection. VIHD locates visually dominant decoder layers via Visual Dependency Probing (VDP), executes Visual Intervention Decoding (VID) via token masking to calibrate the semantic distribution, and quantifies the resulting Calibrated Semantic Entropy (CSE) as a reliable hallucination signal. Extensive experiments on three medical VQA benchmarks with two medical MLLMs demonstrate that VIHD consistently outperforms state-of-the-art methods, underscoring the importance of fine-grained visual dependency for hallucination detection. The code will be available at https://github.com/Jiayi-Chen-AU/VIHD
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4543_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Jiayi-Chen-AU/VIHD
Link to the Dataset(s)
VQA-RAD dataset: https://huggingface.co/datasets/flaviagiammarino/vqa-rad
SLAKE dataset: https://huggingface.co/datasets/BoKelvin/SLAKE
VQA-Med-2019 dataset: https://huggingface.co/datasets/claudioreeves/imageclef-vqa-med-2019
BibTex
@InProceedings{CheJia_VIHD_MICCAI2026,
author = { Chen, Jiayi AND Ma, Benteng AND Liao, Zehui AND Chong, Winston AND George, Yasmeen AND Cai, Jianfei},
title = { { VIHD: Visual Intervention-Based Hallucination Detection for Medical Visual Question Answering } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a training-free hallucination detection method for medical VQA called VIHD. The method first identifies decoder layers with strong visual dependency, then masks highly attended visual tokens during decoding to generate intervened responses, and finally fuses the original and intervened semantic distributions into a calibrated semantic entropy score for hallucination detection. The main contribution is therefore a targeted visual intervention framework for introspective hallucination detection, together with an adaptive entropy calibration scheme and empirical validation on three medical VQA benchmarks and two medical MLLMs.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Clear methodological novelty. The paper explicitly identifies visually dominant decoder layers, performs attention-guided token masking to produce intervened responses, and uses the resulting response shift to calibrate semantic entropy.
2.Strong empirical performance. The method consistently outperforms all compared baselines across three datasets, two backbones, and two metrics.
3.Useful ablation and analysis. The paper studies the contribution of each component (VDI, VID, CSE), compares different layer-selection and masking strategies, analyzes sliding-window width and masking ratio, and shows robustness under a different sampling strategy (Top-K).
4.Good fit to the medical MLLM setting. The motivation is clinically relevant: hallucinated but fluent answers are a real concern in medical VQA. The method is training-free and does not rely on external verifiers at inference time, which makes it attractive compared with heavier verification pipelines.
5.Generally clear presentation. The pipeline is easy to follow, Fig. 1 is helpful, and the equations map cleanly to the algorithmic components.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Detection requires repeated stochastic decoding (M=10 for normal responses and M=10 for intervened responses), semantic clustering with DeBERTa-Large-MNLI, and access to internal attention and visual tokens. This may limit practical deployment. A runtime or memory analysis is missing.
2.Novelty is meaningful but somewhat incremental relative to recent introspective/perturbation-based work. The paper is clearly related to prior semantic-uncertainty and intervention-style methods. The contribution is a well-designed refinement rather than a completely new paradigm, so the paper should position this distinction a bit more explicitly.
3.Some analyses are still missing. I would have liked to see statistical significance or variance over runs, dedicated close-ended results rather than only open-ended/all, qualitative examples of true positives/false positives, and failure-case analysis. These would help readers understand when the intervention signal is most reliable.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I lean toward weak accept because the paper addresses an important problem in medical MLLMs, proposes a technically sound and reasonably novel method, and demonstrates consistent improvements over strong baselines across multiple datasets and backbones. However, I do not rate it more highly because the novelty, while meaningful, is somewhat incremental relative to recent work, and the computational overhead of the detector is not analyzed.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes the VIHD framework, a novel method for hallucination detection in medical MLLMs. The approach introduces VDI to identify key layers with significant visual dependency, enabling partial visual token masking during the decoding process to perturb visual representations. Additionally, the authors propose calibrated semantic entropy to measure semantic dispersion between normal responses and visual-intervened responses. Comprehensive experiments on three distinct datasets and two medical MLLMs demonstrate the effectiveness of the VIHD method. Furthermore, detailed ablation studies validate the necessity and contribution of each key component in the framework.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper provides a clear and detailed introduction to existing hallucination detection methods, offering readers a comprehensive overview of the current landscape. 2.The proposed method is innovative, introducing VDI and VID to inject perturbations into key visual tokens of critical latent layers. Additionally, CSE effectively evaluates semantic dispersion, providing a reliable and robust indicator for hallucination detection. 3.The experimental evaluation is thorough. The main results demonstrate significant performance gains compared to existing state-of-the-art methods. Furthermore, the detailed ablation studies and parameter sensitivity analyses reveal the stability and robustness of the VIHD framework.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Minor comment: 1.The names VDI and VID are very similar, which may easily confuse readers. The authors might consider using more distinctive names for these designs to enhance readability. 2.The authors can consider reporting the time cost of the proposed method, as computational efficiency is an important factor in evaluating methods for hallucination detection.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This work addresses hallucination detection, a critical challenge in MLLMs. The authors present VIHD, a novel and effective approach that combines key visual token perturbation with semantic entropy computation for identifying hallucinations. The experimental results are robust and thoroughly validated. Therefore, I recommend accepting this paper.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes VIHD, a training-free, introspective pipeline for detecting hallucinations produced by medical multimodal LLMs in VQA. Rather than treating the MLLM as a black box and perturbing its inputs, VIHD reaches inside the decoder with three coupled stages. Visual Dependency Identification (VDI) aggregates cross-modal attention from generated tokens to visual tokens layer by layer and selects a sliding window of w consecutive layers whose cumulative visual attention is maximal. Visual Intervention Decoding (VID) then re-runs generation while masking the top-ρ highest-attention visual tokens inside those visually dominant layers, producing an “intervened” response. Finally, Calibrated Semantic Entropy (CSE) draws M normal and M intervened responses, clusters them into semantic equivalence classes with DeBERTa-MNLI, and fuses the two distributions either complementarily (when the cosine similarity ≥ τ) or contrastively (otherwise); the entropy of the fused distribution is the hallucination score. On VQA-RAD, SLAKE, and VQA-Med-2019 with Hulu-4B and Lingshu-7B, VIHD improves AUC over the strongest prior method (VASE) by 1.68–12.00 points.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The conceptual pitch is clear and genuinely distinct from existing introspective detectors: VASE and Semantic Entropy rely on heuristic input-level perturbations, whereas VIHD performs a targeted internal intervention on the very visual tokens the model is attending to, which is a cleaner operationalisation of “is the answer actually supported by visual evidence.”
2.The empirical study is broad— three benchmarks, two recent medical MLLMs, seven baselines, two metrics (AUC and AUG), and a sampling-strategy robustness check
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The originality of the approach is weak. The central mechanism — masking high-attention visual tokens and contrasting the resulting distribution with the original — is close in spirit to Visual Contrastive Decoding (Leng et al., CVPR 2024 [14]) and Self-Introspective Decoding (Huo et al., ICLR 2024 [9]), both of which the paper cites but does not compare against as detection methods.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
VIHD proposes a clean, training-free internal-intervention pipeline (VDI + VID + CSE) that consistently outperforms seven strong baselines across three medical VQA datasets and two MLLMs with well-structured ablations, but its novelty is incremental over VASE / Semantic Entropy / Visual Contrastive Decoding and its key hyperparameters appear tuned on the test splits without a held-out validation set, placing it just above the acceptance bar — hence weak accept.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely thank the reviewers and ACs for their time, effort, and insightful comments. We provide point-by-point responses to their remaining concerns below.
R1Q2 & R3Q1: Novelty of VIHD Existing introspective hallucination detection approaches treat MLLMs as black boxes, relying on external input perturbations and measuring response stability. Such designs capture only output consistency, without revealing whether the generated response is grounded in visual evidence. In contrast, VIHD probes the internal cross-modal causal dependencies between generated tokens and visual inputs. Specifically, we identify visually dominant decoder layers and perform targeted visual token masking to construct intervention-based detection signals. This shifts the detection criterion from output stability to internal grounding fidelity.
Regarding VCD and SID, both are designed for hallucination mitigation, aiming to suppress hallucinations during generation. In contrast, VIHD focuses on post-hoc hallucination detection, assessing the hallucination risk of a generated response. This distinction in objective and evaluation protocol makes VCD and SID not directly comparable to VIHD. Therefore, they are excluded from the main experiments.
R1Q1 & R2Q2: Time cost The inference time of VIHD is approximately 11.59 seconds per sample, which is comparable to the best competing method VASE (11.55 seconds per sample). This indicates that VIHD’s detection performance is achieved without additional computational overhead.
R1Q3: Further analysis Thanks for the constructive comments. Due to page limits, we will incorporate these analyses in the extended journal version.
R2Q1: Module name Thanks for the useful suggestion. To avoid confusion with Visual Intervention Decoding (VID), we will rename Visual Dependency Identification (VDI) as Visual Dependency Probing (VDP) in the final version.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
While the reviewers note some concerns regarding incremental novelty and computational overhead, there is a consistent positive consensus that the paper addresses a critical and timely problem in medical multimodal LLMs and proposes a technically sound, well-motivated, and practically useful solution. Given all positive scores, I recommend acceptance at this stage but authors need to address the concerns in their final version.