Abstract

Vision-language alignment using chest X-rays and radiology reports has emerged as an advanced paradigm for zero-shot classification and grounding of chest X-ray findings. However, standard contrastive learning typically treats radiographs and reports from different patients simply as negative pairs. This assumption introduces noisy negatives, as different patients frequently exhibit similar findings. Such noisy negatives cause semantic ambiguity and degrade performance in zero-shot understanding tasks. To address this challenge, we propose CoNNS, a concept-guided noisy-negative suppression framework. To support the negative suppression mechanism, unlike previous methods that use raw reports or templatized texts, we construct a hierarchical concept ontology using large language models. The ontology structures 41 key clinical concepts by explicitly modeling presence, attributes (location and characteristics), and texts (evidential segment and presence statement). Leveraging this ontology, we implement a cross-patient pair relabeling strategy comprising three steps: (1) Fine-Grained Breakdown to categorize pairs based on finding presence; (2) Noisy Negative Filtering to resolve semantic conflicts by removing false negatives; and (3) Hard Negative Mining to identify subtle attribute discrepancies using a lightweight language model. Finally, we propose a Concept-Aware NCE loss to align visual features with text while suppressing the identified noisy negatives. Extensive experiments across multi-granularity zero-shot grounding tasks and five zero-shot classification datasets validate that CoNNS outperforms existing state-of-the-art models. The code is available at https://github.com/DopamineLcy/conns.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0377_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/DopamineLcy/conns

Link to the Dataset(s)

N/A

BibTex

@InProceedings{LiaChe_ConceptGuided_MICCAI2026,
        author = { Lian, Chenyu AND Zhou, Hong-Yu AND Wong, Chun-Ka AND Qin, Jing},
        title = { { Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    Standard contrastive learning in radiographs and reports treats image–text pairs from different patients as negatives. This assumption introduces noisy negatives, since different patients can share the same findings. The paper studies this issue and shows that such pairs are incorrectly pushed apart despite being semantically consistent. To address this, the authors propose a concept-guided framework that converts reports into a structured ontology and uses it to relabel cross-patient pairs, filtering false negatives and identifying harder ones. They further introduce a concept-aware contrastive loss that leverages these refined relations to improve alignment, leading to better zero-shot classification and grounding.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper addresses a real limitation of contrastive learning in medical vision-language models by identifying and mitigating noisy negatives across patients, introducing a concept-level formulation that better reflects semantic consistency.

    • The proposed framework is coherent and enables more accurate supervision and improved alignment by combining a structured ontology, cross-patient pair relabeling, and a concept-aware contrastive loss.

    • The experimental results show consistent improvements over strong baselines across tasks, supporting the effectiveness of the approach.

    • The paper is well written and clearly structured, making the method easy to follow.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The method critically depends on LLM-based extraction of structured concepts from radiology reports, which is used to relabel training pairs and guide supervision. However, the paper provides no quantitative or clinical validation of this ontology. Since datasets such as MIMIC-CXR do not provide ground-truth structured annotations of this form, it remains unclear how accurate or reliable these extracted labels are, and errors in this stage could propagate to the learning objective.

    • The ontology is defined by a fixed set of 41 target entities (as specified in the extraction prompt), but the paper does not explain how this concept set was constructed. It is unclear whether these concepts are manually curated, derived from existing clinical ontologies, or adapted from prior work (e. g. , [A]). Could the authors clarify the selection process for these 41 concepts and discuss how sensitive the method is to this choice?

    • While the ablation study shows that the full system improves performance, it does not clearly disentangle the contributions of ontology-based text representation and noisy-negative suppression, making it unclear which component is primarily responsible for the observed gains. In particular, the ablation evaluates components built on top of the ontology, but does not assess the design, coverage, or impact of the ontology itself.

    [A] Peng, Yifan, et al. “Automatic recognition of abdominal lymph nodes from clinical text. “ Proceedings of the 3rd Clinical Natural Language Processing Workshop. 2020.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a relevant problem and proposes a framework that consistently outperforms baselines. However, I have some concerns about the methodology, particularly the reliance on an unvalidated LLM-generated ontology by a medical expert and the lack of clarity around the concept set (see Weaknesses). For now, my rate is “weak accept”, but I hope authors could clarify these points in the rebuttal.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    1.the authors identify the critical issue of noisy negatives in contrastive learning for chest X-ray vision-language alignment. By proposing a concept-guided suppression framework, they effectively reduce semantic ambiguity and improve the reliability of zero-shot classification and grounding tasks.

    2.the authors construct a hierarchical clinical concept ontology and integrate it into a multi-step relabeling strategy. This design enables fine-grained reasoning over findings and attributes, leading to more precise alignment and strong performance gains across multiple datasets.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.the authors identify the critical issue of noisy negatives in contrastive learning for chest X-ray vision-language alignment. By proposing a concept-guided suppression framework, they effectively reduce semantic ambiguity and improve the reliability of zero-shot classification and grounding tasks.

    2.the authors construct a hierarchical clinical concept ontology and integrate it into a multi-step relabeling strategy. This design enables fine-grained reasoning over findings and attributes, leading to more precise alignment and strong performance gains across multiple datasets.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Major Weaknesses: 1.The rationale behind the design of the 41 predefined concepts is not clearly explained. In Section 2.2, “N is the number of sampled texts per image” are unclear. it is not specified whether the text is sampled directly from the original reports or from an external source. In addition, the definition of M_{i,j} is ambiguous. The meaning of this variable and the reason why its shape is (NB) \times B are not sufficiently justified. It is unclear why the formulation is not simply N \times B. Furthermore, the interpretation of the entries in M_{i,j} is confusing. In particular, the choice of assigning M_{i,j}=1 to represent a “No–No” correspondence, rather than a “Yes–Yes” match, is counterintuitive and requires further explanation.

    2.In the concept-aware NCE loss, the use of an arbitrary number of positive pairs has already been explored in prior works such as [1] and [2]. The authors are encouraged to briefly discuss the differences and connections between their approach and these existing methods, in order to better highlight the novelty and contribution of the proposed formulation.

    [1] Stablerep: Synthetic images from text-to-image models make strong visual representation learners, NeurIPS, 2024.

    [2] Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation, CVPR, 2025.

    3.To the best of my knowledge, the Open-I dataset only provides image–report pairs and does not include explicit classification labels. It is therefore unclear how zero-shot classification is conducted on this dataset.

    Minor Weaknesses: 1.In Section 2.1, the statement that “each [sample] is sampled with a 50% probability during training” is unclear in relation to the official data split of the MIMIC-CXR dataset.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    see the weaknesses

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    CoNNS is a concept-guided framework to suppress noisy negatives in chest X-ray vision-language alignment. Unlike standard contrastive learning that treats all cross-patient pairs as negatives, CoNNS constructs a hierarchical concept ontology from radiology reports using an LLM, introduces a cross-patient pair relabeling strategy (fine-grained breakdown, noisy negative filtering, and hard negative mining), and proposes a Concept-Aware NCE loss that aligns visual and text features while masking out ambiguous pairs. Extensive experiments on zero-shot grounding and classification tasks show consistent improvements over prior state-of-the-art methods.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Identifies and addresses a critical limitation of standard contrastive learning in medical imaging – the noisy negatives problem where cross-patient pairs with similar findings are incorrectly treated as negatives – and proposes a principled concept-based suppression framework. 2.Introduces a novel hierarchical concept ontology paired with a three-step cross-patient relabeling strategy, which preserves both linguistic diversity and prompt consistency while enabling fine-grained discrimination of findings. 3.Demonstrates strong empirical gains over state-of-the-art methods across multiple zero-shot grounding (word, phrase, sentence levels) and five zero-shot classification datasets, with ablation studies confirming the contribution of each component.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Reliance on a predefined and fixed set of 41 clinical concepts – The hierarchical ontology is limited to these concepts, which may not cover all possible findings or generalize to other diseases or imaging modalities without manual re-construction. 2.Potential scalability and computational overhead – The framework involves LLM-based ontology construction, a lightweight LM for hard negative mining, and a concept-aware attention mechanism, which adds complexity and training cost compared to simpler contrastive baselines.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses a meaningful problem (noisy negatives in chest X-ray contrastive learning) with a novel concept-guided suppression framework, and shows solid empirical gains. However, the approach relies on a fixed set of 41 concepts, adds computational complexity, and is only validated on chest X-rays. Given the trade-off between novelty/performance and limited generality, a Weak Accept is appropriate.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers and meta-reviewer for the positive assessment and constructive suggestions. We would like to clarify several points here:

1.The 41 concepts are predefined chest X-ray finding phrases, which help identify semantically consistent, contradictory, and ambiguous cross-patient pairs. Their selection is based on previous chest X-ray datasets and discussions with radiologists.

2.Extracting structured information from chest X-ray reports during training is a common practice in prior work, such as CARZero and RadZero. Although this step may introduce noise, it has been shown to perform better than using raw reports or the original auto-extracted labels in MIMIC-CXR.

3.Sampled texts refer to the texts sampled from the selected concepts, as well as the texts generated by counterfactual prompting. The shape (N·B) * B indicates that the contrastive learning is conducted with N·B texts and B images in a batch.

4.Regarding the Open-I dataset, we follow the labels used in the CARZero codebase for our zero-shot classification experiments.

5.LLM extraction is an offline preprocessing step, and the lightweight NLI model is only used for pending “Yes-Yes” pairs during training; inference does not require these extra modules.

6.Prior work, such as MedCLIP, partially mitigates the issue of false negatives by leveraging finding labels, but it still overlooks the rich and subtle information in radiology reports.

We have revised the camera-ready version to clarify key points within the limited space of the paper. The code will be made publicly available to facilitate further details and discussion.




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    All reviewers recommend WA, motivated by the importance of the problem, namely false negative handling in contrastive vision-language pre-training for radiology, and the strong empirical performance compared to prior pre-training strategies. The concerns raised are minor and mainly relate to positioning with respect to prior work and how the ontology set is defined. In particular, the paper would benefit from a clearer and more complete discussion of related methods for entity extraction, positive pair refinement, and hierarchical knowledge processing in radiology VLM pre-training, such as MedCLIP, KAD, SAT (TMI 2023), CXR-CLIP (MICCAI 2023), DLILP (IPMI 2025), ASG (MICCAI 2024), and other related works. These points can be addressed in the final version.



back to top