List of Papers Browse by Subject Areas Author List
Abstract
Vision–language models (VLMs) show promise for clinical decision support in radiology because they enable joint reasoning over radiological images and clinical text, thereby leveraging complementary clinical information. However, radiological findings are long-tailed in practice, leaving some conditions underrepresented and making zero-shot inference essential. Yet current CLIP-style medical VLMs are sensitive to prompt variations and often lack trustworthy external knowledge at inference time, which hinders reliable clinical deployment. We present \textit{KEPIL}, a prompt-robust framework that integrates curated medical knowledge to stabilize zero-shot generalization. KEPIL comprises: (i) \emph{dynamic prompt enrichment} using ontologies with LLM assistance, (ii) a \emph{semantic-aware contrastive loss} aligning embeddings of equivalent prompt variants via a dual-embedding objective, and (iii) \emph{entity-centric report standardization} to yield ontology-aligned representations. Across seven benchmarks, KEPIL achieves state-of-the-art zero-shot/finetuning performance in classification and segmentation; under prompt-variation tests, it improves AUC by (6.37\%) on \textit{CheXpert} and by (4.11\%) on average. Ablations and qualitative analyses validate the contributions of enriched prompts and semantic alignment, while attention maps highlight clinically relevant regions. These results suggest that structured knowledge and robust prompt design are key to clinically reliable radiology-facing VLMs. Code will be released at \texttt{***}
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1810_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Roypic/KEPIL
Link to the Dataset(s)
MIMIC-CXR Dataset: https://physionet.org/content/mimic-cxr/2.1.0/
CheXpert Dataset: https://stanfordmlgroup.github.io/competitions/chexpert/
ChestXray-14 / NIH Chest X-ray Dataset: https://nihcc.app.box.com/v/ChestXray-NIHCC
PadChest Dataset: https://bimcv.cipf.es/bimcv-projects/padchest/
RSNA Pneumonia Detection Challenge Dataset: https://www.kaggle.com/competitions/rsna-pneumonia-detection-challenge
SIIM-ACR Pneumothorax Segmentation Dataset: https://www.kaggle.com/competitions/siim-acr-pneumo
BibTex
@InProceedings{LuoHao_KEPIL_MICCAI2026,
author = { Luo, Haozhe AND Shu, Shelley Zixin AND Zhou, Ziyu AND Berke, Robert AND Reyes, Mauricio},
title = { { KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detection } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes KEPIL, a radiology vision-language framework for prompt-robust zero-shot disease detection.
Its central idea is to ground the text side of zero-shot inference in selected medical knowledge rather than relying only on fixed prompts or unconstrained LLM-generated descriptions.
Concretely, the submission combines:
- entity-centric report standardization using RadGraph,
- knowledge-enriched prompts built from sources such as UMLS, Radiopaedia, and the Fleischner glossary with LLM-assisted normalization,
- a knowledge query module that couples visual features and enriched textual entities, and
- a semantic-aware contrastive objective that encourages invariance across prompt variants. The paper also contributes a relatively broad empirical study across seven thoracic imaging benchmarks, including seen, unseen, rare-disease, and cross-modality settings.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The paper addresses a timely and practically important problem. Prompt sensitivity is a obstacle for zero-shot/open-vocabulary deployment of medical VLMs, and the paper frames this issue clearly and convincingly.
-
The use of selected knowledge resources is a meaningful design choice. Grounding prompt enrichment in UMLS / Radiopaedia / Fleischner-style descriptors is more clinically defensible than relying solely on free-form LLM generations.
-
The empirical section is broad. The model is tested on seven public datasets spanning seen diseases, unseen diseases, rare diseases, and a CXR-to-CT transfer setting.
-
The prompt-robustness experiments are valuable. Evaluating prompts produced by multiple LLMs and including realistic perturbations such as typos, omissions, and punctuation errors moves beyond standard zero-shot testing and better matches actual user behavior.
-
The ablation studies are useful and mostly well targeted. The paper examines the effects of enriched prompts, the semantic-aware contrastive loss, dropout choices, placement of the loss, and the degree of textual detail at inference time.
-
Overall, the method is clearly: the knowledge pipeline, report preprocessing, prompt construction, and robustness objective all support the same high-level claim
-
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The novelty is somewhat incremental relative to prior radiology VLM literature. KAD already emphasizes knowledge-enhanced report/image pretraining, MedKLIP injects medical knowledge into the language-image pipeline, CARZero studies cross-attention alignment for radiology zero-shot classification, and MAVL decomposes disease descriptions for richer textual supervision. KEPIL is a sensible synthesis and extension of these ideas, but the conceptual jump over those baselines is moderate rather than decisive.
-The claimed mechanism for prompt-variant alignment does not match the actual loss formulation. The abstract presents a key contribution as “aligning embeddings of equivalent prompt variants via a dual-embedding objective. ” However, the semantic-aware contrastive loss L_scdescribed in the method is applied to two dropout-perturbed views of the same report embedding, not directly to multiple semantically equivalent prompt variants. In other words, the implemented objective appears closer to embedding consistency regularization than explicit prompt-variant alignment. This weakens the claimed novelty and raises concerns about whether the proposed mechanism truly addresses prompt robustness in the way the paper describes
-The claim of state-of-the-art performance is inconsistent with the paper’s own results in Table 1. In the section on seen-disease classification, the manuscript states that KEPIL “consistently achieves the highest performance in terms of both AUC and ACC across all five datasets. ” However, Table 1 does not support this claim. On CheXpert, KEPIL reports AUC = 91.21, which is lower than CARZero = 92.38, and its ACC = 87.88 is also not the highest because DeViDe = 87.98.This is a serious issue because the central conclusion of the paper is not fully supported by its own reported numbers.
-
The manuscript claims both classification and segmentation performance, but the experiments mainly demonstrate classification only. The paper states that KEPIL achieves strong performance in “zero-shot classification and segmentation. ” However, the reported evaluation metrics are primarily AUC, F1, and ACC, which are classification-oriented metrics. In particular, SIIM-ACR is a segmentation dataset, yet the paper reports classification-style results without clearly explaining how segmentation labels were converted into classification targets, and without reporting standard segmentation metrics such as Dice or IoU. This creates a substantial mismatch between the claimed contribution and the actual empirical evidence
-
The experimental protocol is not reproducible enough for a method whose core contribution depends on LLM-based knowledge processing. Reproducibility is currently only moderate. The submission says code will be released upon acceptance, but key details remain underspecified: The manuscript states that ChatGPT-4o is used for prompt standardization and completion, and even claims that it performs best as shown in Table 3. However, the paper does not provide sufficient details about the actual prompting pipeline, system instructions, ontology constraints, normalization rules, or post-processing procedures. Since the LLM-based knowledge generation is a central component of the framework, the lack of implementation detail makes the method difficult to reproduce and independently verify. The situation is made worse by the placeholder statement “Code will be released at ***. ”
-
The “open-vocabulary” and “clinical reliability” claims should be toned down or better justified. The benchmarks are all thoracic datasets, and the model is still evaluated on a fixed set of disease categories per dataset. This is useful, but it is narrower than true open-world clinical deployment.
-There are serious citation and reference problems that undermine academic reliability. Table 1 includes a baseline called “DeViDe [?]”, which clearly indicates a missing or unresolved citation. In addition, the experimental settings cite [12] for components such as BioClinicalMPBERT weights and M3AE pretraining on MIMIC, but reference [12] in the bibliography is actually CARZero, not the source of those components. These are not minor formatting issues; they are significant scholarly errors involving incorrect or missing references for key methodological components
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My overall recommendation is Weak Accept.
I view the paper as marginally above the acceptance threshold because it tackles an important and underexplored problem in medical VLMs, presents a knowledge-grounded approach, and backs the method with a broad empirical study that includes prompt-variation robustness rather than only standard zero-shot evaluation.
Hower, the methodological novelty over the closest prior art is moderate, some of the strongest claims are broader than what is actually demonstrated, and reproducibility/reporting should be improved. The authors should use the rebuttal and final revision to clarify the novelty relative to KAD / MedKLIP / CARZero / MAVL, correct the segmentation overclaim, and add missing implementation details.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes KEPIL, a knowledge-enhanced vision–language framework aimed at improving zero-shot disease detection in radiology with an emphasis on robustness to prompt variations. The method combines (i) ontology-grounded prompt enrichment using curated sources (Radiopaedia, UMLS, Fleischner glossary) with LLM-assisted normalization, (ii) a “semantic-aware” contrastive loss on a lightweight text adapter to enforce invariance to perturbations, and (iii) a Knowledge Query Module that performs token-level cross-attention between visual patches and textual entity descriptions. Across seven public datasets, KEPIL reports state-of-the-art or competitive zero-shot performance for seen, unseen, and rare conditions, and demonstrates reduced sensitivity to prompt variability.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Integrates curated, verifiable medical knowledge bases with LLM assistance to produce structured, clinically grounded prompts, reducing hallucination risk compared to LLM-only approaches.
2.Introduces a simple but effective adapter-level contrastive consistency objective to stabilize text representations without unfreezing the backbone.
3.Employs a token-level cross-attention “Knowledge Query Module” to better connect fine-grained visual cues with radiographic descriptors.
4.Entity-centric report parsing with RadGraph helps standardize supervision and aligns training signals with ontology entities.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.How exactly are ontology constraints enforced during LLM-based standardization (e.g., prompts, schema, validation/repair rules)? Please include examples and acceptance criteria that prevent hallucinated attributes from entering the knowledge base.
2.The Knowledge Query Module is a standard cross-attention formulation; novelty rests more in system integration than in the core modeling primitives.
3.Recent and closely related radiology VLMs focused on robust open-vocabulary alignment and grounding are not fully discussed.
4.Several baseline names in prompt-robustness plots (e.g., MAE, C2C2, EAD, MeanCLIP) are unexplained in the main text, making it hard to interpret those comparisons.
5.For the LIDC-IDRI CT evaluation, how are prompts tailored to CT findings (e.g., nodule attributes) and how do you handle 3D context with 2D slices? Could you compare to CT-specific VLMs or report the limitations explicitly?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
KEPIL addresses an important and timely problem—prompt-robust, ontology-grounded zero-shot disease detection in radiology—and demonstrates strong empirical gains across multiple public datasets and realistic prompt-variation tests. While some concerns should be addressed before the acceptance.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes a knowledge-enhanced radiology vision–language framework, KEPIL, for more robust zero-shot chest X-ray classification. Its main contribution is to combine three ideas into a single training and inference pipeline: (1) ontology-guided prompt enrichment using curated medical knowledge and LLM assistance, (2) a semantic-aware contrastive objective that encourages embeddings of equivalent prompt variants to stay aligned, and (3) entity-centric report standardization so that training text is mapped into a more structured, ontology-aligned representation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed method goes beyond simple prompt enrichment by using LLM-generated knowledge to create multiple semantically equivalent prompts and enforcing their alignment through a contrastive objective, while also standardizing report-derived supervision via ontology-based entity extraction.
First, the paper addresses a real and important weakness of current medical VLMs: prompt instability. Second, the method is thoughtfully structured. Third, the empirical evaluation is fairly broad. Fourth, the work appears competitive against strong recent baselines.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main gains may stem from prompt consistency regularization and structured supervision, while the contribution of LLM-based knowledge enrichment itself is less clearly isolated. My main concern is that the paper’s framing around “long-tail” and “zero-shot generalization” is somewhat stronger than what the experiments fully establish. Relatedly, the paper should be clearer about what exactly makes a disease “unseen” versus merely drawn from a different dataset. A second weakness is that parts of the method are not as novel as the paper’s presentation may suggest. A third weakness is reproducibility. A fourth weakness is that the paper’s interpretability/qualitative evidence is hard to assess from the current presentation. Finally, although the authors argue clinical reliability, the paper still evaluates benchmark classification performance rather than downstream clinical workflow utility.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My recommendation is driven by the fact that this is a solid and relevant paper with a reasonable technical design, meaningful empirical gains, and a clear focus on an important practical issue: prompt robustness in radiology VLMs. At the same time, I am not fully convinced by the strength of the paper’s broader claims around long-tail zero-shot generalization.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
N/A
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Three reviewers agree that this paper addresses a critical and timely problem in medical vision-language models, prompt robustness, by proposing a thoughtfully designed KEPIL supported by extensive empirical evaluations. While the integration of curated medical ontologies and the realistic prompt-robustness experiments are highly commended, the reviewers identified several issues regarding overclaimed novelty, methodological clarity, and manuscript preparation. I am recommending a Provisional Accept, conditional upon the authors thoroughly addressing the reviewers’ feedback in their final revision.
