List of Papers Browse by Subject Areas Author List
Abstract
Medical vision-language models (MedVLMs) have progressed rapidly and demonstrate strong performance across a broad range of clinical tasks, including report generation, disease classification, and cross-modal retrieval. The success can be attributed to contrastive language-image pretraining (CLIP), which holistically aligns image and text into a shared embedding space, thereby unifying visual and linguistic representations. While these models excel on coarse-grained medical tasks, where global image-level semantics are sufficient for success, such as modality recognition, disease classification, and standard image-report retrieval, it remains challenging for them to succeed in fine-grained and evidence-driven clinical scenarios. In those settings, models are supposed to identify and retrieve specific diagnostic evidence and align it to clinically applicable criteria. Otherwise, misidentifying or overlooking diagnostic evidence may lead to clinically contradictory errors and weaken the reliability and interpretability of the final diagnoses. Therefore, rigorously quantifying CLIP’s fine-grained retrieval capability in realistic clinical settings is essential. In this paper, we introduce WHO-CXRBench, a benchmark grounded in WHO chest X-ray (CXR) diagnostic guidelines for fine-grained diagnostic rule retrieval. We curate an evaluation set of 6{,}115 patient-level samples, resulting in 29{,}375 patient-rule pairs. We quantify the retrieval performance by Recall@K, Precision@K, nDCG, and mAP. Experiments show that even the best-performing baseline achieves only 14.23\% Recall@10 and 13.06\% nDCG@10, indicating that current CLIP-style MedVLMs still have space for improvement in fine-grained diagnostic rule retrieval. We believe that WHO-CXRBench builds a valuable foundation for evaluating MedVLMs’ performance under realistic clinical settings. Code and dataset are available at https://github.com/LyxDLiI/WHO-CXRBench
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0505_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/LyxDLiI/WHO-CXRBench
Link to the Dataset(s)
MIMIC-CXR-JPG dataset: https://www.physionet.org/content/mimic-cxr-jpg/2.1.0/
MIMIC-Ext-MIMIC-CXR-VQA: https://physionet.org/content/mimic-ext-mimic-cxr-vqa/1.0.0/
BibTex
@InProceedings{LiuYix_WHOCXRBench_MICCAI2026,
author = { Liu, Yixiang AND Cheng, Pujin AND Cai, Zhiyuan AND Huang, Yijin AND Wang, Zhonghua AND Xu, Zhongxing AND Tang, Xiaoying},
title = { { WHO-CXRBench: A CXR Benchmark for Diagnostic Rule Retrieval in CLIP-Style MedVLMs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
Deciphered request for structured English summary frameworkMajor Contributions. WHO-CXRBench introduces the first WHO-guideline-grounded benchmark specifically designed for fine-grained diagnostic rule retrieval in chest X-rays, filling a clear gap left by existing coarse-grained MedVLM evaluations. The paper proposes a three-stage automated construction pipeline — Prior-Guided Content RAG for hierarchical guideline extraction, GRADE-RA for quality-controlled rule filtering, and report-based personalized rule alignment — that reduces a noisy 1,154-rule corpus to a curated 300-rule set while preserving all 37 disease categories. Additionally, a VQA rationality check demonstrates that the extracted personalized rules serve as meaningful clinical context, improving downstream diagnostic performance.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strengths. The problem motivation is compelling and clinically grounded: the “clinical contradiction” failure mode in existing CLIP-style retrieval is illustrated concretely and resonates with real-world radiology practice. The benchmark pipeline is principled and largely reproducible, with GRADE-RA offering a transparent, deterministic quality scoring mechanism. The evaluation is comprehensive, covering 12 state-of-the-art CLIP-style MedVLMs across both general-purpose and CXR-specialist categories, and the finding that broader medical-domain pretraining (UniMed-CLIP) outperforms CXR-specialist models is a genuinely interesting and actionable insight for the community.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Weaknesses. The most critical concern is that the core task assumption — that image-to-rule retrieval is a valid proxy for clinical reasoning — is never substantively justified with clinical or cognitive science evidence. Methodologically, the LLM judge used for positive label construction is never identified or ablated, creating a reproducibility gap and potential circular validation risk; the τ = 0.5 alignment threshold lacks task-specific sensitivity analysis; and the manual audit’s 92.7% precision figure is inflated by cherry-picking top-confidence rules rather than random sampling. Beyond methodology, the paper’s analysis is shallow: it reports that all models perform poorly but offers almost no explanation of why — no failure case analysis, no breakdown by disease category or rule frequency, and no discussion of what architectural changes would actually improve performance on this benchmark.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The overall score reflects a paper with a genuinely valuable research direction that is, at this stage, insufficiently substantiated to merit acceptance. The benchmark addresses a real and underexplored problem — the disconnect between semantic similarity optimization in CLIP-style MedVLMs and the evidence-driven, guideline-grounded nature of clinical radiology — and the three-stage construction pipeline demonstrates meaningful engineering effort. However, two factors weigh heavily against acceptance in the current form. First, the foundational assumption that retrieval of WHO diagnostic rules is a clinically valid proxy for radiological reasoning is never defended beyond intuition; without this justification, the entire benchmark’s clinical relevance remains an assertion rather than an established fact. Second, the methodological transparency is inadequate for a benchmark paper, where reproducibility is paramount: the undisclosed LLM judge, the unjustified threshold τ, and the biased manual audit collectively undermine confidence in the quality of the ground-truth labels — which are, after all, the benchmark’s most critical artifact. On top of these concerns, the experimental analysis stops at reporting numbers rather than generating insight, which limits the paper’s contribution to the field. A strong benchmark paper should not only reveal that models fail, but explain the structure of that failure well enough to guide future model design. With substantial revisions addressing these points, the work has clear potential; in its current state, it falls short of the standard expected for MICCAI publication.Sonnet 4.6Extended
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper introduces WHO-CXRBench, a benchmark grounded in WHO chest X-ray (CXR) diagnostic guidelines for fine-grained diagnostic rule retrieval. Experiments indicate that current CLIP style MedVLMs still have space for improvement in fine-grained diagnostic rule retrieval. The proposed WHO-CXRBench builds a valuable foundation for evaluating MedVLMs’ performance under realistic clinical settings.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The proposed WHO-CXRBench addresses a clear gap in existing evaluation protocols by focusing on diagnostic rule retrieval rather than standard image–report matching or classification. This shift toward knowledge-grounded evaluation is conceptually valuable and could encourage more clinically relevant model development. 2.The benchmark is constructed based on WHO diagnostic guidelines, which enhances its clinical credibility. The inclusion of structured diagnostic rules and patient-level alignment provides a more interpretable and verifiable evaluation target compared to conventional free-text retrieval tasks. 3.The experimental results reveal that existing CLIP-style MedVLMs perform poorly on fine-grained diagnostic rule retrieval (e.g., low Recall@K and nDCG), providing useful evidence that current models are insufficient for evidence-level reasoning. This diagnostic insight is valuable for the community. 4.The inclusion of VQA-based validation and manual inspection of rule alignment strengthens confidence in the constructed benchmark and supports its practical relevance.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The reported poor performance of existing models is a central claim of the paper; however, it is unclear whether this is primarily due to model limitations or the intrinsic difficulty and potential noise of the proposed benchmark. In particular, errors introduced during rule construction and report–rule alignment may contribute to the observed performance degradation. The paper does not sufficiently disentangle these factors, making it difficult to draw definitive conclusions. 2.Several key steps in the pipeline—including rule extraction, quality assessment, and report–rule alignment—depend heavily on LLM-based processing. This raises concerns regarding:
- reproducibility,
- potential hidden biases,
- error propagation across stages. While the authors provide some validation (e.g., manual audit), a more systematic analysis of noise sensitivity and robustness would strengthen the credibility of the benchmark. 3.The inclusion of MRM in Table 3 is questionable, as it is not a CLIP-based method, unlike the other compared approaches. This inconsistency may lead to unfair or less meaningful comparisons and should be clarified or revised to ensure methodological alignment across baselines.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed WHO-CXRBench provides a challenging evaluation setting for both general-purpose and CXR-specialized CLIP-style models, highlighting an important gap in fine-grained, evidence-based retrieval. However, the paper lacks sufficient clarity regarding several key aspects of the benchmark design and experimental settings, which makes it difficult to fully assess the validity and fairness of the evaluation. These points should be carefully addressed in the rebuttal.
Overall, the work is well-motivated and potentially valuable, but further clarification is needed to strengthen its credibility.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have addressed most of my concerns in the rebuttal, so it can be accepted to as a poster.
Review #3
- Please describe the contribution of the paper
This paper introduces WHO-CXRBench, a new benchmark designed to evaluate medical vision-language models (MedVLMs) on the task of fine-grained diagnostic rule retrieval. Motivated by the gap between coarse-grained semantic alignment evaluation and the actual evidence-driven workflow of clinicians, the authors construct a dataset based on the WHO chest radiography manual and MIMIC-CXR.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The authors provide a comprehensive comparison of 12 state-of-the-art CLIP-style MedVLMs, spanning both general-purpose models and CXR-specialist models. 2.The inclusion of a Visual Question Answering (VQA) rationality check proves that the retrieved rules actually serve as useful contextual evidence, significantly improving Qwen2.5-VL-7B’s performance. 3.The paper identifies a weakness in current AI models. The results demonstrate that contrastive learning optimized for image-report similarity fails to capture localized evidence-to-rule grounding.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The LLM generates conf(r,g) to define \mathcal{G}_\mathcal{S}^+. LLM-based relevance judgment introduces semantic bias into ground truth, skewing evaluation toward biased models. 2.GRADE-RA boosts quality but sharply reduces data scale. The final corpus \mathcal{G} has only 300 rules across 37 diseases (8.1 per disease), too limited to capture real radiological complexity. 3.Personalized golden rules use a fixed threshold \tau=0.5.Diverse, nuanced clinical language makes this rigid cutoff exclude valid low-confidence matches or admit marginal false positives.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposes a valuable WHO-CXRBench for fine-grained diagnostic rule retrieval in medical vision-language models, with comprehensive SOTA model comparisons, valid VQA rationality verification, and insightful findings on the flaws of current contrastive learning designs. However, critical limitations remain: LLM-based ground truth introduces semantic bias, the final rule dataset is overly small to reflect real radiological complexity, and the fixed threshold for golden rules is rigid for nuanced clinical language. These merits support acceptance while the non-trivial flaws prevent a higher score, leading to my overall recommendation of Weak Accept.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The authors have partially responded to reviewer concerns in the rebuttal but failed to resolve all core critical issues. Solved points include supplementing LLM model information, $\tau$ threshold ablation analysis, partial failure case analysis, architectural ablation results, removing inconsistent baseline MRM, and clarifying that VQA results do not cause circular validation.
Nevertheless, several fundamental flaws remain unaddressed. First, systematic analysis of LLM-induced semantic bias and error propagation is incomplete, and full-scale human verification is unfinished, leaving reproducibility and ground-truth credibility concerns unresolved. Second, the limited rule corpus size still cannot fully reflect real radiological complexity, and sufficient justification for its adequacy is lacking. Third, the core assumption that image-to-rule retrieval serves as clinical reasoning proxy still lacks solid clinical and cognitive evidence, only with wording revision. Fourth, it still cannot clearly disentangle performance degradation caused by model limitations or benchmark inherent noise. Besides, the inflated manual audit precision issue is not explained at all.
Author Feedback
Q1.LLM construction (reproducibility, bias, and error propagation) [R1&R2-2&R3-1]. We agree that LLM-assisted construction raises risks. For reproducibility, the omitted LLM construction details are now provided in the anonymized repository. Code and benchmark will be released upon acceptance. For hidden bias, we used a rule-matching strategy that feeds the LLM one report and one rule at a time to judge whether they match, rather than having it select a rule from a preset set. This trades computational cost for accuracy via semantic understanding. Experiments confirmed that different LLMs make stable decisions in most cases. A cross-LLM report-rule matching check between DeepSeek-R1-0528-Qwen3-8B and Llama-3.1-8B-Instruct achieved 86.8% binary agreement, suggesting that most report-rule support decisions are stable across LLMs. For error propagation evaluation, following MedR-Bench (NC, 2025), we have started human verification of the LLM-extracted rule corpus. Manual auditing of all report-rule matches is being expedited, with 1543/6513 cases completed. We will add stage-wise analysis for rule extraction, GRADE-RA filtering, and report-rule alignment in the final version. Q2.τ sensitivity [R1&R2-2&R3-3]. This analysis was omitted due to space limits. τ=0.3 gives broader coverage but adds 37013 low-confidence pairs in the 0.3–0.5 range, reaching 73448 total pairs and 10.8 rules per case. τ=0.5 offers a relatively balanced trade-off, with 5.8 rules per case and 36435 pairs. τ=0.7 yields higher confidence but reduces rule diversity, with 3.0 rules per case and 14283 pairs. The VQA task performed best at τ=0.5.Combining these results with clinicians’ qualitative feedback, we selected τ=0.5.This is also consistent with recent LLM ranking works using 0.5 as the relevance-score threshold, e.g., G-RAG (arXiv, 2025) and Synthetic-Data Reranking (RANLP, 2025). Q3.Size and failure analysis [R1&R2-1&R3-2]. Given our focus on high-precision, image-verifiable rules and LLM hallucination risks, our strict filtering trades diversity for reliability. Meanwhile, rule count alone does not fully capture benchmark diversity, as a single rule may encompass multiple findings, locations, and conditions. We will add rule-length statistics to clarify this point. Failure analysis on UniMed-CLIP reveals non-uniform errors. Disease-level failures are more frequent in categories with overlapping CXR patterns (e.g., pneumonia and lung neoplasms), where opacities, consolidation, or infiltrates overlap, making qualifiers like laterality and distribution critical. At the rule level, longer and more compositional rules are harder; rules in the upper length quartile (≥23 words) achieve only 7.3% Recall@10, compared to 18.2% for shorter rules. Q4.Architecture and MRM [R1&R2-3]. For architecture, stronger visual encoders improve performance. MedCLIP improved from ResNet50 to ViT, increasing Recall@10 from 1.85% to 8.27%, and UniMed-CLIP improved from ViT-Base to ViT-Large, increasing Recall@10 from 9.62% to 14.23%. In contrast, enlarging UniMed-CLIP’s text encoder decreased Recall@10 to 9.35%, suggesting that visual grounding matters more than generic text-side scaling. For MRM, we agree with R2-3 and will remove it from the comparison. Q5.Circular validation and VQA utility [R1]. VQA is not used to validate ground-truth correctness. Instead, it only tests whether retrieved rules provide useful downstream context to an independent VLM, Qwen2.5-VL-7B. Ground-truth quality is assessed separately by manual audit against CXR and report evidence, with a full audit ongoing. Thus, construction, VQA testing, and quality audit are separated. Q6.Clinical and cognitive scope [R1]. We apologize for any lack of clarity in our original description. As shown in Fig. 1, our goal is to evaluate an intermediate step in clinical diagnosis; specifically, whether models can map raw CXR images to guideline-based diagnostic criteria. We will revise the wording accordingly.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents WHO-CXRBench, a WHO-guideline-grounded benchmark for fine-grained diagnostic rule retrieval from chest X-rays, evaluating 12 CLIP-style MedVLMs. The benchmark addresses a critical gap in coarse-grained MedVLM evaluation, is clinically grounded in WHO guidelines, and includes a comprehensive comparison of models plus a VQA rationality check showing retrieved rules improve downstream performance. However, all reviewers highlight heavy LLM dependence for rule extraction, quality filtering, and ground-truth alignment—raising semantic bias, reproducibility, and error-propagation concerns without sufficient sensitivity analysis. The final rule set is too small (300 rules across 37 diseases, ~8 per disease) to capture real radiological complexity. Analysis remains shallow: no failure-case breakdown by disease category or rule frequency, and the paper does not disentangle whether poor performance stems from model limitations versus benchmark noise.
Issues to be addressed in rebuttal:
- LLM-based ground-truth construction: identify the LLM, analyze bias/error propagation, and justify its use without circular validation.
- Small rule set size: discuss whether 8.1 rules per disease is sufficient for fine-grained retrieval; consider sensitivity to data scale.
- Shallow analysis: add failure case examination, performance breakdown by disease/rule frequency, and architectural insights.
- Provide clinical/cognitive justification: core assumption that image-to-rule retrieval proxies clinical reasoning.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The authors’ rebuttal partially addresses several methodological concerns raised by the reviewers, including disclosing the LLM details, providing τ threshold sensitivity analysis, adding preliminary failure case breakdowns, reporting architectural ablation results, removing the inconsistent MRM baseline, and clarifying that the VQA experiment does not constitute circular validation. However, critical issues identified in the original reviews remain insufficiently resolved. Most significantly, the rebuttal does not provide a systematic analysis of LLM‑induced semantic bias and error propagation across the three‑stage pipeline; the human verification of report‑rule matches is reported as incomplete. Whether the rule corpus size (300 rules, ~8.1 per disease) is limited continues to lack adequate justification for capturing real‑world radiological complexity, with only a promise to add rule‑length statistics rather than substantive evidence. Finally, the rebuttal does not disentangle whether poor model performance stems from benchmark noise versus genuine model limitations. Given these foundational concerns, the paper does not meet the standards for acceptance.
A particularly controversial issue, raised by Reviewer #1 and echoed here, is that the authors implicitly assume that image‑to‑rule retrieval proxies clinical reasoning without clinical or cognitive validation. The ground‑truth rules are aligned and derived directly from the report (the final diagnosis), and any intermediate diagnostic evidence not mentioned in the report would be excluded from the positive set. Consequently, the retrieval task is reduced to predicting a subset of the final diagnosis rather than capturing an independent intermediate reasoning step. This conflation means the benchmark does not truly assess whether a model can retrieve clinically meaningful intermediate evidence; it simply tests alignment with the documented diagnosis. The authors have not provided any clinical validation that the retrieved rule set, constructed in this report‑dependent manner, corresponds to actual clinician reasoning or guideline‑application steps. Without such validation, the benchmark’s claimed clinical grounding is undermined. Combined with the incomplete human verification and unresolved bias from LLM‑based label construction, the benchmark is not ready for publication.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper presents retrieval of relevant guideline as a proxy for correct interpretation of chest X-rays in preliminary reads provided by VLM models. The reviews are a bit mixed (appears to have some automation in some of the reviews) and the authors appear to be responding to some of the concerns.
