List of Papers Browse by Subject Areas Author List
Abstract
Reliable deployment of Vision-Language Models (VLMs) in radiology requires validation metrics that go beyond surface-level text similarity to ensure clinical fidelity and demographic fairness. This paper investigates a critical blind spot in current model evaluation: the use of decoding strategies that lead to high aggregate token-overlap scores despite succumbing to template collapse, in which models generate only repetitive, safe generic text and omit clinical terminology. Unaddressed, this blind spot can lead to metric gaming, where models that perform well on benchmarks prove clinically uninformative. Instead, we advocate for lexical diversity measures to check model generations for clinical specificity. We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports. Weighted Association Erasure (WAE) aggregates these shifts to measure the clinical signal loss across demographic groups. We show that deterministic decoding produces high levels of semantic erasure, while stochastic sampling generates diverse outputs but risks introducing new bias, motivating a fundamental rethink of how “optimal” reporting is defined.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2453_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://pypi.org/project/clinical-cad/
Link to the Dataset(s)
N/A
BibTex
@InProceedings{ParAdi_Measuring_MICCAI2026,
author = { Parikh, Aditya AND Feragen, Aasa AND Das, Sneha AND Frank, Stella},
title = { { Measuring What VLMs Don’t Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution of this paper is formalizing a vocabulary-level diagnostic framework to evaluate radiology report generation. Specifically, it introduces Clinical Association Displacement (CAD) and Weighted Association Erasure (WAE) to quantify how standard decoding strategies can lead to template collapse. This framework helps reveal how the erasure of critical clinical terminology is often hidden behind high aggregate text similarity scores.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A major strength of this work is its empirical identification of a genuine and often overlooked evaluation blind spot. The authors astutely point out that models can game traditional metrics like BERTScore by producing safe and generic templates. Furthermore, treating the decoding strategy itself as a primary confounding variable is a very insightful experimental design. The sensitivity analysis clearly illustrates the fundamental trade-off between semantic erasure under deterministic decoding and emergent bias under stochastic sampling, providing a strong empirical foundation for their observational claims.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The primary weakness of the paper is its conceptual and methodological incrementalism. While the identified problem is significant, the proposed solution relies on standard statistical techniques. The CAD framework is essentially a direct application of the log ratio of unigram probabilities with Dirichlet smoothing, which is a well-established method in text analysis. Applying this to generated text is a natural extension rather than a fundamentally new methodology. Secondly, the paper lacks direct clinical validation. It remains unproven whether the reference-aligned demographic association shifts measured by CAD and WAE actually correspond to clinically meaningful diagnostic errors or patient harm. Finally, critical reproducibility details are missing, such as the exact patient and site-level data splits and the specific prompts used for the GPT-4.1 baseline.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend a weak reject because there is a notable gap between the importance of the problem identified and the depth of the proposed solution. The paper does an excellent job of highlighting how conventional metrics fail to penalize template collapse. However, the proposed CAD and WAE metrics are conceptually incremental and lack the rigorous clinical validation necessary to justify their adoption as definitive fairness or diagnostic metrics in medical AI. Without reader studies or validation against true clinical outcomes, it is difficult to accept these surrogate metrics as a robust standard, making the overall contribution somewhat limited for a top-tier venue.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
After reading the rebuttal, I appreciate the authors’ clarifications, especially that CAD/WAE are intended as complementary, system-level diagnostics rather than per-sample clinical metrics. However, the rebuttal does not substantially change my assessment.
My main concerns remain. First, the proposed method is still conceptually incremental: it formalizes a useful intuition—checking whether clinically relevant terminology is preserved—but does not convincingly show why CAD/WAE are methodologically necessary beyond simpler clinical term/entity coverage or omission analyses. Second, the clinical validity of the proposed metrics remains insufficiently established. The rebuttal gives intuitive examples, but still does not show that CAD/WAE correlate with diagnostic correctness, radiologist judgment, patient-level errors, or clinical utility. Third, reproducibility remains only partially addressed, since code, prompts, and model weights are promised post-acceptance, while several protocol details remain unclear in the submitted manuscript.
Overall, I agree that the paper identifies an important evaluation blind spot in radiology report generation. However, the rebuttal mainly reiterates the motivation and intended use of the metrics rather than resolving the concerns about novelty, clinical validation, and reproducibility. I therefore maintain my recommendation to reject.
Review #2
- Please describe the contribution of the paper
The paper examines a limitation of current evaluation practices in radiology report generation, where standard text similarity metrics (e.g., BLEU, ROUGE, BERTScore) can assign high scores to outputs that rely on repetitive, generic templates while omitting clinically specific terminology.
To analyze this issue, the authors propose Clinical Association Displacement (CAD), a vocabulary-level framework that measures how the demographic associations of clinical terms (e.g., between male and female reports) shift between reference and generated corpora. They further introduce Weighted Association Erasure (WAE), which aggregates these term-level shifts into a global measure of lexical displacement.
Using these metrics, the paper studies the effect of different decoding strategies and shows that deterministic decoding tends to produce high-scoring but lexically collapsed outputs, while stochastic decoding increases diversity but can introduce new demographic associations. The authors suggest using CAD and WAE as complementary evaluation tools to detect such failure modes and to better distinguish models that produce clinically informative reports from those that achieve high scores through generic generation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper is well-motivated. Direct NLP metrics are not clinically representative, and high overlap scores can mask clinically uninformative outputs. This is a valid and important issue in medical report generation.
2.The proposed idea to detect such cases of collapsing through analyzing what the model does not say is interesting and novel.
3.The results are informative and align well with intuitive expectations. They show that deterministic decoding yields high evaluation scores but leads to severe template collapse and loss of clinical terminology, while stochastic decoding improves diversity at the cost of increased bias and instability. This trade-off is consistently demonstrated and can provide useful insights into model behavior for future work.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Corpus-level analysis only: CAD/WAE operate at the dataset level and do not provide per-sample insight. In contrast to standard metrics (e.g., BERTScore, BLEU), which can be evaluated per sample, the proposed metrics are not directly applicable at the individual report level. This limits their usefulness for analyzing specific model errors or for practical clinical validation.
2.Unclear link to clinical correctness: While the paper reports text-based metrics (e.g., BERTScore, RateScore) as indicators of clinical performance, it does not establish whether CAD/WAE correlate with true diagnostic accuracy or clinical utility. It remains unclear whether improvements in these metrics correspond to better medical outcomes.
3.Overemphasis on inference: The paper attributes the observed effects primarily to decoding strategy. However, similar phenomena (e.g., template collapse or loss of terminology) may also originate from training dynamics, such as data imbalance or reliance on language priors. This aspect is not sufficiently discussed.
4.Minor presentation issue: Figure 2 layout (subfigure labeling) could be improved for clarity.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My recommendation is to accept. The paper addresses an important and underexplored issue in evaluating radiology report generation with VLMs and provides a diagnostic framework. The findings are clear and informative. However, I encourage the authors to address the points above.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Regarding the corpus-level nature of CAD/WAE, my concern was not that corpus-level analysis is inappropriate. I agree that bias, terminology erasure, and template collapse are naturally system-level phenomena. My concern was rather that, unlike standard metrics such as BLEU, ROUGE, or BERTScore, which can be computed both at the corpus level and at the individual-report level, the proposed metrics operate only at the corpus level. This limits their usefulness for identifying specific erroneous reports or supporting granular clinical validation. However, the authors clarify that CAD/WAE are intended as supplementary system-level auditing tools rather than replacements for standard metrics. This framing is reasonable and largely satisfies my concern.
Regarding clinical validation, I still think the paper should be careful not to overstate the clinical meaning of CAD/WAE. The authors provide plausible examples of clinically relevant terminology erasure, and I agree that a full clinical trial is not necessary for this paper. However, the proposed metrics remain indirect text-level indicators rather than validated measures of diagnostic correctness or patient impact. Therefore, they should be presented as tools for flagging potential clinical risks, not as direct evidence of clinical harm or clinical utility.
Overall, the rebuttal clarifies the intended role of the proposed metrics and addresses my main concern sufficiently. I would maintain my positive recommendation.
Author Feedback
We sincerely thank the reviewers for their time and valuable feedback. We are happy to see their unanimous appreciation of our work and their recognition of the importance of addressing failures of current VLM evaluation metrics. Our framework identified a critical evaluation blind spot in radiology report generation (RRG) where reports collapse to a repetitive template, and introduced metrics which go beyond surface-level text similarity and measure vocabulary erasure and newly biased terms.
Please note - Reviewer 3’s [R3] review text concerned a completely different paper than ours (as it critiques “TAVR-VLM”, which is unrelated to our work). However, [R3’s] comment on Field #9 (additional comments) does address our paper and sounds quite positive (“insightful analysis”, “compelling and much-needed warning for community”). We sincerely hope this will be taken into consideration.
We address criticisms individually -
1.[R1, Meta#1] Incrementality in Solution: R1 correctly notes that our underlying mathematical formulation (log-odds, Dirichlet smoothing) relies on established statistical principles. However, we want to point out that if there exists a simple solution to a problem, it does not seem sound to reject those in favor of more complicated solutions. Our application of tracking vocabulary-level association shifts between human reference and model prediction is indeed new (as [R2] mentioned “interesting and novel”). Our core contribution is a new validation method for RRG, not the statistical formula. Given the increase in RRG studies across all domains, by establishing this methodological baseline, we hope to standardize demographic-association methodology, which is an important aspect of fairness in report generation.
2.[R2, Meta#3] Corpus-level Analysis: We disagree that operating at the corpus level is a limitation. Algorithmic bias, erasure, and template collapse are meant to be measured at the system level, not per-sample: A model cannot be labeled as ‘biased’ or ‘not fair’ based on a single sample. Thus, we recommend using the proposed tools together with standard evaluation metrics (mentioned in Discussion & Conclusion). Our method is intended to be used as a system-level validation method, rather than “real-time or granular error analysis”.
3.[R1, R2, Meta#2] Clinical Validation: The problems we highlight (template collapse, generated reports falsely associating ‘aicd’, ‘pacemaker’ with women, and ‘prostate’ as neutral) are fundamental, and we provide empirical evidence. Identifying the problem at this level does not require expensive, time-consuming clinical trials. We do agree that the mitigation strategies should indeed be validated before use in the clinic. In additional space post-acceptance, we would like to add more qualitative examples showing how CAD-flagged terms directly correspond to self-evident harm (eg, “stenosis” and its CAD-level shift).
4.[R1] Reproducibility: As mentioned in References [1][2] in the abstract, we will provide code, a Python package for the framework, prompts, and model weights post-acceptance. To eliminate ambiguity, we confirm the use of identical prompt templates across all experimental settings.
5.[R2]: Overemphasis on Inference: We agree that training dynamics may also influence the observed phenomenon, and we test it with an imbalanced dataset. We will discuss this further. Thank you for pointing this out.
6.[R2] Minor Presentation Issue: We are happy to improve the labels in Fig 2. The reviewers identified our motivation and recognized the importance of this research in VLM evaluation. We appreciate their praise for our experimental design and analysis of decoding trade-offs. We believe this paper provides a fundamental rethink of validation in report generation. Our auditing tool can help move past the ‘illusion of performance’ and ensure safe and equitable deployment in medical AI. We again thank the reviewers for their appreciation and feedback.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper identifies a research gap in current radiology report generation evaluation: models that achieve high surface-level similarity scores (e.g., BLEU, BERTScore) often suffer from template collapse, where they produce safe, generic text while erasing specific clinical terminology. To quantify this, the authors introduce Clinical Association Displacement and Weighted Association Erasure. These metrics measure demographic-based word association shifts and clinical signal loss, specifically highlighting how deterministic decoding strategies can lead to the erasure of critical diagnostic details across different patient populations.
While the paper presents good contributions, its recommendation for acceptance is contingent upon the authors clarifying the following points during the rebuttal phase:
1.Reviewers noted that the core methodology, using log ratios of unigram probabilities with Dirichlet smoothing, is a standard statistical technique in text analysis. While applying these to the specific problem of “terminology erasure” in radiology is useful, it is viewed as an application of existing methods rather than a fundamentally new methodological breakthrough.
2.A primary concern is the missing link between the proposed metrics (CAD/WAE) and actual medical outcomes. It remains unproven whether a high “Association Erasure” score directly correlates with diagnostic errors, patient harm, or decreased clinical utility. Provide evidence or a discussion on how shifts in CAD/WAE correlate with specific diagnostic omissions (e.g., missing a “severe aortic stenosis” finding).
3.Unlike standard metrics like BERTScore, which provide a score for every individual report, CAD and WAE are corpus-level metrics. They provide insights into dataset-wide trends but cannot identify errors in a specific patient’s report. This limitation restricts their use for real-time model auditing or granular error analysis.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The proposed corpus level evaluation additional to other report level measurement is a valuable measure for model evaluation. Reviewer 3’s comments have been updated and considered.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
Unfortunately, I have to recommend reject despite an interesting effort in a modern and popular subject.
I noticed that there were two opposite outcomes recommended after the rebuttal round; so, I reviewed the the paper myself (I have relevant expertise).
I found the main claim that the common metrics incentivize VLMs to generate templated radiology reports to be supported poorly. The paper never explicitly explains how template bias would actually emerge during the training. Besides, the metrics (BLEU, ROUGE, BERTScore) are largely obsolete in this field, with radiology report evaluation having long moved to domain-specific metrics like RadGraph and RadCliQ (thus, the “gaming” argument was not convincing). I also found the empirical analysis to be somewhat superficial (for example, comparing greedy vs. beam search decoding and using output diversity to validate the proposed metric is obviously not sufficient). Lastly, I also have to agree with the post-rebuttal comments of R1.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper identifies an important evaluation blind spot in radiology report generation: standard metrics fail to penalize template collapse. The proposed CAD and WAE provide a novel vocabulary-level diagnostic framework to quantify this issue. The analysis of decoding strategies reveals a valuable trade-off between semantic erasure and emergent bias. While Reviewer #1 raises concerns about incrementalism, Reviewer #2 and Reviewer #3 finds the work well-motivated and informative. The rebuttal clarifies that CAD/WAE are system-level auditing tools, not per-sample metrics. Recommendation: Accept.
