List of Papers Browse by Subject Areas Author List
Abstract
Medical Large Vision-Language Models (Med-LVLMs) achieve strong capability but remain prone to hallucinations that pose serious diagnostic risks. Retrieval-Augmented Generation (RAG) has been explored to mitigate this issue. However, existing approaches append retrieved evidence as text to the language input, which shifts the model’s focus toward textual cues instead of visual reasoning and increases the risk of copying retrieved reports. To address this, we propose a RAG framework that leverages retrieved image–text pairs to enhance the visual representation of a query image through modified cross-attention. We incorporate a novel training objective into this module to prevent over-reliance on a few references and promote semantically faithful alignment. In addition, we observe that commonly used CLIP-based retrieval often assigns high similarity to negated and non-negated sentences with contradictory clinical meanings, resulting in mismatched references and potential hallucinations. To mitigate this, we design a negation-aware retrieval strategy that filters clinically inconsistent cases prior to feature fusion. Experiments on three benchmarks in radiology and ophthalmology demonstrate consistent improvements over the base model and text-prompt RAG baselines, by reducing language shortcuts and reinforcing visual evidence utilization. Source code will be released upon publication.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3169_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{KimYuj_Rethinking_MICCAI2026,
author = { Kim, Yujoong AND Kim, Soopil AND An, Sion AND Park, Sang Hyun},
title = { { Rethinking RAG: Mitigating Text-Driven Hallucinations via Feature-Level Retrieval Integration in Medical LVLMs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a medical RAG framework that integrates retrieved image-text pairs at the feature level rather than appending retrieved reports as text prompts. The core idea is to inject retrieved evidence into the vision encoder through gated cross-attention, so retrieval can improve visual grounding without encouraging direct text copying. The paper also introduces a grounding loss to prevent attention collapse onto only a few retrieved references and a negation-aware retrieval strategy to filter clinically contradictory cases before fusion. Compared with standard text-prompt RAG, this is an interesting reformulation because it explicitly targets the language-shortcut problem in Med-LVLMs. Overall, the main contribution is a more visually grounded retrieval integration strategy for reducing text-driven hallucinations in medical LVLMs.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
First, the paper addresses a meaningful and timely problem, namely that conventional medical RAG may unintentionally increase hallucinations by biasing generation toward retrieved text instead of image evidence. Second, the proposed feature-level retrieval integration is technically interesting and reasonably novel in the medical LVLM setting, especially when combined with the attention grounding loss and the negation-aware filtering module. Third, the method is supported by a fairly comprehensive evaluation across three benchmarks spanning both radiology and ophthalmology, where it consistently outperforms the base model and several strong RAG baselines such as FactMM-RAG, RULE, and MMed-RAG. Fourth, the ablation study is useful and shows that both the grounding loss and negation-aware retrieval contribute substantially to performance. Finally, the attention-ratio analysis is a particularly strong aspect of the paper because it provides mechanistic evidence that the method indeed increases reliance on visual tokens rather than only improving final metrics.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main weakness is that the novelty is moderate rather than fully groundbreaking, since the work still builds on the broader medical RAG, while the cross-attention style fusion is also conceptually related to prior multimodal architectures. In particular, the negation-aware retrieval module is effective but somewhat heuristic, and simple sign inversion of embeddings, and the paper does not compare this against stronger learned semantic contradiction modeling. A second weakness is that the evaluation still relies mainly on VQA metrics and ROUGE/METEOR, without a stronger clinical factuality analysis or expert reader study to directly verify hallucination reduction in practice. In addition, the VQA benchmark is constructed from GPT-4-generated yes/no pairs, which introduces an additional synthetic layer and may not fully reflect real clinical questioning settings. Overall, these limitations keep the paper at a weak accept level for me rather than a clear accept, but the idea is still promising and the empirical evidence is solid enough to merit positive consideration.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I give a weak accept / borderline accept recommendation because the paper addresses an important limitation of current medical RAG systems and proposes a well-motivated feature-level retrieval integration strategy that is supported by consistent improvements across multiple benchmarks. The method is not fully groundbreaking, as it builds on an already active line of medical RAG research, and the clinical validation could be stronger, but the technical idea is clear, the analysis is thoughtful, and the empirical results are strong enough to make the paper a worthwhile contribution.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The authors present a Retrieval-Augmented Generation (RAG) framework designed to mitigate “text-driven hallucinations” in Medical Large Vision-Language Models (Med-LVLMs). Recognizing that standard RAG methods append retrieved reports as text—thereby causing models to bypass visual reasoning in favor of copying textual cues—the proposed method fuses retrieved image-text pairs directly into the visual representation via a gated cross-attention mechanism inserted into the vision encoder. To support this, the authors introduce a correspondence-based grounding loss that prevents attention collapse, and a negation-aware retrieval strategy that inverts the CLIP embeddings of negated sentences to filter out clinically contradictory cases. Evaluated on IU X-ray, MIMIC-CXR, and Harvard-Fair VLMed, the method demonstrates superior VQA and report generation performance while empirically increasing the model’s reliance on visual tokens.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The proposed method is generally well-motivated and properly validated. Instead of directly appending the retrieved reference pairs as raw images/texts, the paper fused them into the original query image token using a properly designed cross-attention mechanism before entering the LLM. The experimental results in Table 4 further support the argument, where the proposed method indeed aligns the LLM’s attention to the image tokens.
2.The idea of using the gated fusion mechanism and grounding loss is quite elegant and intuitive. The gated fusion mechanism ensures that the fused features are combined with the raw ViT feature adaptively and ensures the worst-case performance. And the grounding loss aligns the local correlation between the query patches and reference images with their semantical similarity.
3.The evaluation in the paper is very detailed and robust. It not only demonstrates a uniform performance improvement on three different datasets with two different modalities but also provides a thorough ablation that analyzes the contribution of each building component.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.One of the reviewer’s major concerns is the design of negation-aware retrieval. Using negation detection to avoid contradictory retrieved pairs is intuitively convincing. Directly negating the feature embedding is not that straightforward. “Mass presented” and “Mass not presented” are not opposite in terms of high-dimensional geometry. Additionally, naive negation detection may also confuse cases like “Mass presented” and “Mass free”, where both are affirmative sentences but their meaning are completely different.
2.The complexity of the proposed method is quite concerning, given its marginal performance improvements. The proposed method introduces multiple new hyperparameters like $alpha$ and $K$. However, the sensitivity with respect to these hyperparameters is not properly evaluated in the paper (limited by the paper length). On the other hand, the performance improvements in Table 1 compared with the baselines are not very significant as well. Most of the improvements are only ~1.5% to the second-best method.
3.The base model used here is relatively outdated. The paper uses LLaVA-Med-1.5 as the base model, but this is a model that was released 3 years ago, and there are more robust base models like Lingshu, MedGemma, and HuatuoGPT-V. It would be more helpful to evaluate the proposed method on more recent baselines.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.The proposed method helps to shift the VLM’s attention towards the vision tokens according to Table 4.But is there any theoretical explanation for this? The proposed method should have the same token length as the raw LLaVA-Med baseline, but why does the proposed method give a much higher visual-token attention score?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, this is a solid paper with enough contribution to the field. While there are still some issues to be answered, it does not harm its quality to pass the acceptance bar.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
- A Multimodal RAG framework for Med-VLMs that integrates retrieved image–report pairs directly into the vision encoder via gated cross-attention, rather than appending retrieved evidence as additional text context to the LLM, to prevent language shortcuts and encourage more visual grounding.
- A negation-aware sentence-level retrieval strategy that addresses the well-known insensitivity of CLIP-based similarity to fine-grained clinical semantics, particularly negated findings, by inverting embeddings of negated sentences and applying consistency-based filtering to exclude clinically contradictory cases.
- A correspondence-based attention grounding loss that supervises cross-attention weights using patch-level visual similarity to prevent attention collapse onto a narrow subset of retrieved items.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors clearly identify a fundamental structural limitation in existing text-prompt RAG approaches, language bias and shortcut copying, and address it through a principled feature-level integration. The attention ratio analysis provides convincing empirical support for this motivation.
- The proposed method achieves SOTA results across all benchmarks. This suggests the framework generalizes well across imaging modalities.
- Addressing CLIP’s insensitivity to negated clinical findings is a clinically meaningful and underexplored contribution. The embedding polarity-flip is an elegant and lightweight solution, and the qualitative example effectively illustrates its practical impact in filtering diagnostically contradictory cases.
- Each proposed component is individually validated, and the results demonstrate that all components contribute meaningfully, with the grounding loss and negation-aware retrieval showing the largest individual impact.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Relying solely on NLP metrics (ROUGE-L, METEOR) is inadequate for medical report generation. These metrics measure lexical overlap rather than clinical accuracy. Incorporating clinical efficacy metrics would more convincingly demonstrate the medical validity of generated reports. Currently, you can’t conclude from the report generation table.
- Restricting VQA evaluation to binary yes/no questions risks inflating performance gains due to class imbalance or decision boundary biases. More fine-grained question types (e.g., open-ended, multiple-choice) would provide a more rigorous and clinically meaningful assessment of the method’s diagnostic reasoning capability.
- All experiments are conducted solely on LLaVA-Med. Given that different VLMs exhibit varying sensitivities to additional context, evaluating on at least one additional backbone would strengthen the generalizability claims of the framework.
- Table 3 does not specify which dataset the ablation results correspond to, making it difficult to contextualize the reported numbers.
- Limited architectural novelty: using cross-attention for multimodal feature fusion is a well-established paradigm. In recent multimodal RAG literature (e.g., read Abootorabi et al., Ask in Any Modality, 2025), cross-attention-based fusion has been widely adopted. The contribution is therefore more of a careful domain adaptation than a fundamentally novel architecture, though the specific design choices are well-suited to the medical setting.
- A qualitative visualization of attention patterns, both with and without the loss, is necessary to assess whether the model is attending to semantically meaningful regions.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
Section 2.3 could be written more clearly. Try to make the tables more self‑explanatory.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses a well-motivated limitation of text-prompt RAG in medical VLMs, and the proposed components, particularly negation-aware retrieval and the grounding loss, are clinically informed with consistent empirical improvements. However, architectural novelty is limited, as cross-attention-based multimodal fusion is well-established in the broader literature. Evaluation credibility is further weakened by reliance on lexical metrics alone, binary-only VQA, and single backbone testing. However, the contribution remains valuable to the MICCAI community for its practical insights and design choices. They need some important revisions for the camera-ready if this paper is accepted.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely appreciate the reviewers for their constructive comments, and appreciate the opportunity to clarify several points raised in the reviews. (R1) Negation-aware retrieval is heuristic, just a sign inversion of embeddings: Sign inversion in cosine-similarity space was chosen as a deliberate lightweight and training-free design, broadly compatible with any CLIP-based retriever and free of additional supervision. A fully learned semantic-contradiction model could in principle capture finer cases such as “mass present” vs. “mass free”, but would introduce training overhead and dataset-specific supervision. Combined with the symmetric sentence-level matching (Eq. 1) and the consistency score (Eq. 2), our filtering helps filter clinically contradictory cases in practice, as shown in Fig. 2 (left) and the +5.36pp Acc contribution in Table 3.(R1, R3) Reliance on lexical metrics and binary VQA: These protocols were adopted to align exactly with the medical-RAG baselines we compare against (RULE, MMed-RAG, FactMM-RAG), which all use the same metrics and yes/no VQA, so a controlled comparison required following the same setup. To complement these protocols, we additionally report the attention-decomposition analysis (Table 4), which provides mechanistic evidence that our gains stem from increased visual grounding rather than text copying. Incorporating clinical-efficacy metrics and open-ended VQA would offer complementary evidence in subsequent studies. (R2) Why does the proposed method give a much higher visual-token attention score, even though the input token length is the same as raw LLaVA-Med: The input token sequence to the LLM (system prompt + visual tokens + question) is indeed identical to the raw LLaVA-Med-1.5, so the increase in Vision ratio is not due to changes in token composition. Rather, retrieved evidence is injected into the ViT hidden states via gated cross-attention before the projection, so the visual tokens entering the LLM carry higher task-relevant signal density while their length remains unchanged. As the LLM finds more diagnostically useful information within the visual segment, attention shifts toward it, as shown in Table 4 where the Vision ratio rises from 0.06 to 0.27 on MIMIC-CXR. In contrast, text-prompt RAG appends retrieved evidence as post-vision text tokens, which inflate Post-Vision and further suppress the Vision ratio below the base model. (R2, R3) Experiments are limited to a single and outdated backbone (LLaVA-Med-1.5): The choice of LLaVA-Med-1.5 was intentional, since all medical-RAG baselines we compare against (FactMM-RAG, RULE, MMed-RAG) use the same backbone, which is necessary to isolate the contribution of our retrieval-integration design. Extending the framework to more recent Med-LVLM backbones is a meaningful direction for follow-up work. (R3) Section 2.3 clarity, Table 3 dataset, and table readability: We appreciate these suggestions and agree that these aspects could be presented more clearly. For reference, the Table 3 ablation results are on MIMIC-CXR (the Full model row matches the corresponding row in Table 1).
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
All three reviewers independently assigned a score of 4, showing clear consensus. The paper addresses a well-motivated problem—language shortcuts in text-prompt RAG for medical VLMs—and proposes a coherent feature-level integration via gated cross-attention, a negation-aware retrieval strategy, and a correspondence-based grounding loss. Empirical results are consistent across three benchmarks and two modalities, and the attention-ratio analysis provides mechanistic support for the core claim. Common concerns across reviews include reliance on lexical metrics (ROUGE/METEOR) without clinical efficacy measures, binary-only VQA evaluation, and experiments limited to a single and outdated backbone LLaVA-Med. Authors should address these issues in the camera-ready version.
