Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Interpretable medical image diagnosis requires predictions grounded in clini-cally meaningful findings that physicians can inspect and correct. While Con-cept Bottleneck Models (CBMs) route predictions through predefined imaging signs, they treat concepts as independent scalar scores and struggle to align fi-ne-grained visual features with clinical semantics. We present ConceptAlignE-G, a framework that integrates vision-language model (VLM) based concept alignment with graph reasoning to bridge this gap. Built on BiomedCLIP, a concept-aware perception module grounds each imaging sign in a dedicated visual-semantic embedding space, enabling fine-grained alignment between sign-specific visual features and expert-defined text embeddings. The predict-ed sign probabilities are then used to construct a dynamic sign-image relation-al graph, where a GCN incorporates broader image context beyond the concept bottleneck for disease classification. On CT-based appendicitis differentiation and ultrasound-based breast cancer diagnosis, ConceptAlignE-G outperforms both black-box classifiers and existing concept-based methods in diagnostic AUC and sign recognition. Beyond accuracy, it provides per-sign prediction probabilities and interpretable decision weights, and supports clinician-driven intervention at inference time.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4734_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/MedcAILab/med-cag

Link to the Dataset(s)

N/A

BibTex

@InProceedings{HuYil_Interpretable_MICCAI2026,
        author = { Hu, Yilan AND Xie, Haoyu AND Liang, Dan AND Wei, Xinhua AND Zhou, Hui AND Zhou, Hong AND Huang, Bingsheng},
        title = { { Interpretable Medical Image Diagnosis via VLM-based Concept Alignment and Graph Reasoning } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16886},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes ConceptAlignE-G, an interpretable medical image diagnosis framework that combines VLM-based concept alignment with graph reasoning. The method first uses BiomedCLIP to align image features with predefined clinical signs through sign-specific concept-aware perception modules, and then builds a dynamic sign-image graph where predicted sign probabilities serve as edge weights for GCN-based disease classification. The framework is evaluated on two tasks: CT-based appendicitis differentiation and ultrasound-based breast cancer diagnosis. The paper argues that the method improves both diagnostic performance and interpretability by providing sign-level probabilities and allowing clinician intervention at inference time.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses an important and highly relevant problem for MICCAI: moving beyond black-box predictions toward clinically meaningful and editable explanations. The focus on radiological signs is well motivated and better aligned with clinical workflow than standard saliency-based explanations. 2.The proposed framework is reasonably well designed. In particular, the idea of learning sign-specific visual subspaces via the concept-aware perception module is interesting, and the use of predicted concept probabilities to drive a graph-based diagnostic layer is a sensible extension beyond standard concept bottleneck models. 3.The paper includes experiments on two different tasks and modalities, including an external test set for the appendicitis task. This is stronger than evaluating only on a single internal dataset. 4.The interpretability mechanism is more actionable than post-hoc visualization methods. The concept intervention examples are a valuable part of the paper, since they show how correcting concept predictions can directly change the final diagnosis without retraining.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The experimental validation is still somewhat limited. The breast ultrasound dataset is small, and the paper does not provide enough evidence about robustness, such as confidence intervals, variance across folds/runs, or more detailed statistical analysis beyond a limited DeLong test claim. 2.The baseline comparison is not fully convincing. The proposed method is built on BiomedCLIP, whereas the black-box and concept-based baselines appear to use ResNet50 backbones. This makes it difficult to disentangle the gain from the proposed method itself versus the gain from a stronger pretrained encoder. Stronger foundation-model-based baselines would be needed for a fairer comparison. 3.The “graph reasoning” component is relatively simple. The graph is essentially a star-shaped structure with one image node and concept nodes, and the aggregation resembles weighted fusion of concept embeddings into the image representation. As a result, the technical novelty of the graph module may be viewed as incremental rather than substantial. 4.The interpretability evaluation is mostly qualitative. While the case studies are nice, the paper lacks more systematic evidence that the explanations are faithful, clinically useful, or preferred by experts. For example, there is no quantitative evaluation of intervention success rate, concept calibration, or expert assessment of explanation quality.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I find the paper promising and relevant, with a clear motivation, a reasonable methodological design, and encouraging results across two medical imaging tasks. The sign-level interpretability and intervention mechanism are especially appealing. However, I am not fully convinced by the current empirical evidence, mainly due to the fairness of the baseline setup, limited quantitative interpretability evaluation, and the relatively modest technical depth of the graph reasoning component. Overall, I see this as a bweak accept paper: interesting and potentially valuable, but still needing stronger experimental support to justify a higher score.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper introduces ConceptAlignE-G, an interpretable framework for medical image diagnosis that uses clinically meaningful imaging signs as intermediate concepts.

    The method relies on a dual-encoder setup based on BiomedCLIP, allowing text descriptions of disease signs and associated image features to be mapped into a shared latent space. A concept-aware perception module extracts sign-specific image representations, which are aligned with concept text embeddings to estimate the probability of each sign.

    These sign probabilities are then used to construct a dynamic sign-image relational graph, which is processed by a graph convolution layer to combine image semantics, concept-level evidence, and textual concept knowledge for the final disease prediction. An additional practical aspect is that, because the prediction explicitly depends on intermediate concept scores, a clinician can in principle intervene by correcting concept values at inference time and obtain an updated prediction without retraining.

    Overall, the contribution is a concept-based diagnostic framework that combines sign-specific visual extraction, language-grounded concept alignment, and graph-based aggregation to improve interpretability while maintaining competitive diagnostic performance.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses an important and timely problem in medical AI: improving trust and interpretability in diagnostic models. This is clinically relevant, especially as explainability and accountability are becoming increasingly important for deployment and regulation. 2.The framework provides a more meaningful intermediate explanation than standard saliency or heatmap-based methods. Instead of only indicating where the model attends, it predicts clinically meaningful signs that can be directly interpreted by users. 3.The use of a domain-specific vision-language model, BiomedCLIP, is well motivated. Leveraging text descriptions of imaging signs gives the concept layer semantic grounding and is a sensible way to incorporate medical prior knowledge. 4.The paper is generally well organized and easy to follow, and the method description appears sufficiently detailed for understanding the main pipeline.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The experimental comparison is not fully controlled. The proposed method uses a BiomedCLIP image encoder with a ViT backbone, while the all black-box baselines use ResNet50 (+ DenseNet101). This creates a major methodological confound: it is unclear how much of the gain comes from ConceptAlignE-G itself versus the stronger pretrained backbone and multimodal pretraining. A fairer comparison would include a BiomedCLIP-based black-box classifier with the same, and most likely stronger, image encoder and training protocol, as well as for the CBM/CEM/CCBM-style baselines. 2.The claimed advantage over other concept-based and VLM-based approaches is not yet fully convincing. The paper argues that fixed concept sets create representational bottlenecks, but the proposed method still relies on an explicit set of K binary sign concepts and text embeddings for those signs. In practice, it still routes prediction through a predefined concept interface, so the distinction from existing concept bottleneck approaches should be clarified more carefully, or better explained. 3.The graph reasoning component appears relatively simple. The graph is a star graph centered on the image node, and the paper does not clearly establish why a GCN is preferable to simpler fusion strategies such as concatenation, gated weighting, or MLP-based aggregation. The paper includes ConceptAlignE as an ablation, which however does not target my concer. 4.The breast ultrasound experiment is based on a small dataset (252 images with five-fold cross-validation). That makes claims of superiority less convincing, especially when using a strong pretrained encoder. The paper reports AUC/ACC and mentions DeLong testing, but confidence intervals, fold variance, and broader pairwise statistical testing are missing or incomplete. Given the small sample size and modest margins in some comparisons, stronger statistical reporting is needed. 5.Robustness is not sufficiently characterized. In particular, the paper would benefit from analysis of sensitivity to prompt wording for the text concepts and to concept-label noise, since both are central to the proposed multimodal concept alignment approach. 6.Minor typos are present here and there in the manuscript.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    I encourage the authors to strengthen the experimental section with more controlled and targeted ablations. In particular, the paper would be significantly stronger if it included: (1) a black-box BiomedCLIP classifier with the same image encoder and training protocol; (2) CBM/CEM/CCBM-style baselines built on the same backbone/features; (3) an ablation comparing the proposed GCN-based fusion against simpler alternatives; and (4) robustness experiments for concept-label noise, missing concept labels, and prompt wording.

    The paper would also benefit from stronger statistical reporting, especially for the small breast ultrasound dataset. Reporting confidence intervals, fold-wise variance, and significance testing for the key pairwise comparisons would make the conclusions more convincing.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses an important problem in medical imaging: how to improve interpretability and trust in AI-based diagnosis using clinically meaningful intermediate concepts. The proposed framework is well motivated, clearly presented, and practically relevant. I also find the combination of sign-specific feature extraction, language-grounded concept alignment, and concept-based intervention at inference time to be useful and potentially impactful for clinical AI.

    At the same time, I have reservations about the strength of the empirical evidence supporting the claimed methodological advantages. The comparisons are not fully controlled because the proposed model uses a stronger BiomedCLIP ViT-based encoder while most baselines use ResNet50, making it difficult to isolate the gain from the proposed concept alignment and graph design. In addition, the graph module is relatively simple and the paper does not yet show clearly that it outperforms simpler fusion mechanisms for the right reasons.

    The experimental evidence is also somewhat limited by the small size of the breast ultrasound dataset and by incomplete statistical reporting. The method further inherits a reliance on explicit concept annotations, but the practical impact of this limitation is not deeply examined.

    Overall, I view this as a solid and relevant methodological contribution with good clarity and an interesting interpretable design, but with limited novelty relative to prior concept-based/VLM-based work and with evaluation issues that keep it closer to the acceptance boundary than to a clear accept.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper presents ConceptAlignE-G, an interpretable medical image diagnosis framework that combines vision-language model (VLM) based concept alignment with graph convolutional network (GCN) reasoning. The framework aims to address key limitations in existing concept bottleneck models by introducing sign-specific feature extraction through a concept-aware perception module and incorporating broader image context through graph reasoning. The authors validate their approach on two clinical tasks: CT-based appendicitis differentiation and ultrasound-based breast cancer diagnosis.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Introduces a concept-aware perception (CAP) module that extracts sign-specific visual features for each imaging finding rather than using a shared global representation, enabling more precise alignment between visual features and clinical text descriptions.
    • Constructs a dynamic sign-image relational graph where edge weights are determined by predicted concept probabilities, allowing a GCN to aggregate image semantics, concept text knowledge, and sign predictions jointly for disease classification.
    • The framework naturally supports concept-level intervention at inference time, where clinicians can override incorrect sign predictions and the corrected values immediately propagate through the graph to update the final diagnosis without requiring model retraining.
    • Testing on both CT and ultrasound modalities with different anatomical targets (appendix and breast) demonstrates some generalizability of the approach.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The appendicitis dataset, while multi-center, contains only 1156 patients, and the breast cancer dataset has merely 252 images. These sample sizes are quite small by modern deep learning standards, particularly for a model with multiple components (text encoder, image encoder, CAP modules, GCN). The risk of overfitting is substantial, and it remains unclear how well the approach would scale to larger, more diverse datasets. The reported improvements over baselines, while consistent, are often modest, raising questions about clinical significance.
    • While the intervention mechanism is technically sound, its practical value is unclear. In real clinical workflows, would radiologists actually review per-sign probabilities and manually correct them? The paper provides few anecdotal examples where intervention changes the prediction, but no systematic evaluation of how often radiologists would need to intervene, how difficult it is to identify which concepts to correct, or whether the corrected predictions are actually more accurate.
    • It lacks of sufficient ablation study. Why is a single GCN layer sufficient? Was a deeper graph network explored? The star topology seems restrictive, as it assumes concepts only interact through the central image node rather than allowing direct concept-to-concept relationships that might capture known clinical correlations.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper makes reasonable technical contributions to interpretable medical image analysis and demonstrates consistent, if modest, improvements over baselines. The integration of concept-aware perception with graph reasoning is sensible, and the framework addresses real limitations of existing concept bottleneck models. The validation on two clinical tasks sounds good. However, the work has limited ablation study, poor reproducibility details, smaller dataset experiments for concluding robust generalization and the unvalidated clinical utility of the intervention mechanism.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We appreciate the reviewers’ thoughtful comments. A few clarifications on the major points: On baseline fairness (R1-W2, R2-W1, R2-W2). The concern about backbone mismatch is fair, but BiomedCLIP is not merely a stronger encoder plugged into our pipeline. The CAP module and the graph both depend on the shared vision-language space (Sections 2.2–2.4). Table 1 already shows that stripping the CAP module (BiomedSigLIP) or dropping the GCN for a linear layer (ConceptAlignE) hurts performance within the same BiomedCLIP backbone. This suggests the gains come from the sign-specific perception and graph aggregation, not just the pretrained weights. As for CBM/CEM/CCBM, these methods were built around unimodal CNN encoders; porting them to BiomedCLIP would require redesigning their core feature extractors, which goes beyond a standard baseline comparison. We also want to clarify the distinction from standard CBMs (R2-W2): our CAP module learns sign-specific subspaces instead of projecting one global image feature onto concept vectors, and the GCN fuses image semantics with concept text knowledge and sign probabilities jointly. The text embeddings act as fixed semantic anchors drawn from clinical vocabulary. On the graph module (R1-W3, R2-W3, R3-W3). The star topology mirrors how radiologists actually reason—evaluating each sign relative to the target lesion (Section 2.4). We use a single GCN layer because deeper propagation tends to over-smooth on such a small graph. The ablation in Table 1 supports this: replacing the GCN with a linear layer drops the appendicitis test AUC from 0.713 to 0.685, so the graph structure does add value over simple aggregation. On dataset size and statistics (R1-W1, R2-W4, R3-W1). We agree the breast ultrasound dataset is small (252 images). We ran five-fold cross-validation with DeLong testing; the per-fold breakdown was omitted for space. The appendicitis task at least includes an external test set (n=202, unseen centers), which gives a more meaningful generalization signal than internal validation alone. On interpretability and intervention (R1-W4, R3-W2). The intervention examples in Figure 2c–d are meant to show what the architecture makes possible, not to claim a validated clinical workflow. Real-world utility would need proper human factors studies, and we see that as important follow-up work. On reproducibility (R1, R3). We will release the source code in the near future. Thank you for the careful review.



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    All reviewers acknowledged that the authors are addressing an important problem in medical image analysis. Reviewers also recoganized the contributions in methodology and the use of multiple datasets.



back to top