Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

We introduce CRIMSON, a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules, and prevents normal or clinically insignificant findings from exerting disproportionate influence on the overall score. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists in ReXVal (Kendall’s tau = 0.61–0.71; Pearson’s r = 0.71–0.84), and through two additional benchmarks that we introduce. In RadJudge, a targeted suite of clinically challenging pass–fail scenarios, CRIMSON shows consistent agreement with expert judgment. In RadPref, a larger radiologist preference benchmark of over 100 pairwise cases with structured error categorization, severity modeling, and 1–5 overall quality ratings from three cardiothoracic radiologists, CRIMSON achieves the strongest alignment with radiologist preferences. We release the metric, the evaluation benchmarks, RadJudge and RadPref, and a fine-tuned MedGemma model to enable reproducible evaluation of report generation, all available at https://github.com/rajpurkarlab/CRIMSON.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2039_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/rajpurkarlab/CRIMSON

Link to the Dataset(s)

N/A

BibTex

@InProceedings{BahMoh_CRIMSON_MICCAI2026,
        author = { Baharoon, Mohammed AND Heintz, Thibault AND Raissi, Siavash AND Alabbad, Mahmoud AND Alhammad, Mona AND AlOmaish, Hassan AND Kim, Sung Eun AND Banerjee, Oishi AND Rajpurkar, Pranav},
        title = { { CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16878},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The main contribution of this paper is the introduction of CRIMSON, a clinically grounded LLM-based evaluation framework for generative radiology report assessment. Unlike prior metrics that mainly rely on surface-level similarity, uniform error counting, or coarse significance judgments, CRIMSON explicitly integrates patient context (e. g. , age and indication), finding-level clinical significance weighting, and fine-grained attribute error modeling into a unified evaluation pipeline. Concretely, the framework extracts abnormal findings, assigns severity-aware clinical weights, classifies discrepancies into false findings, missing findings, and eight categories of attribute-level errors such as location, severity, measurement, certainty, and diagnostic over- or under-interpretation, and then computes an interpretable normalized score through a severity-aware scoring formulation. In addition, the paper contributes two new evaluation benchmarks, RadJudge and RadPref, and demonstrates across ReXVal, RadJudge, and RadPref that CRIMSON aligns more closely with expert radiologist judgment than existing report evaluation metrics.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The paper addresses a clinically well-motivated problem and targets an important limitation of existing radiology report evaluation metrics. A major strength is that the paper recognizes that different reporting errors can have very different clinical consequences. Missing a life-threatening abnormality should not be treated the same as omitting a benign or age-related finding. The paper further argues that the importance of a finding may depend on patient age and indication, making the incorporation of patient context into report evaluation both clinically meaningful and practically relevant.
    2. The method shows meaningful novelty in its unified formulation of clinical significance weighting, fine-grained attribute-level error modeling, and context-aware evaluation. CRIMSON not only distinguishes false findings and missing findings, but also models multiple attribute-level discrepancies in matched findings, such as location, severity, measurement, and certainty. At the same time, it introduces clinically motivated significance weighting and allows this significance assessment to depend on patient context. Compared with metrics based mainly on surface similarity or coarse error counting, this unified framework is more aligned with real clinical reasoning.
    3. The treatment of normal findings is a valuable design choice that reflects an understanding of variation in clinical reporting practice. The paper excludes normal findings from the main evaluation on the grounds that reporting of normal findings varies substantially across radiologists, and including them may introduce noise unrelated to true report quality. This is a thoughtful design decision that helps the metric focus more on clinically meaningful content rather than stylistic or template-driven variation.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    One limitation of the paper is that the proposed method is fundamentally an evaluation framework driven by a strong closed-source LLM, with GPT-5.2 explicitly used as the backbone throughout the pipeline. This design choice is not inherently problematic, and prompt design can reasonably be considered part of the methodological contribution. However, it also means that the observed performance may depend, at least in part, on the capability of the chosen judge model. For this reason, comparison against other LLM-based radiology evaluation methods is important. The paper does include at least one clearly LLM-based baseline, namely GREEN, so the gains cannot simply be attributed to the use of an LLM alone; however, the comparison with other closely related LLM-based evaluators is still somewhat limited. More importantly, if the authors intend CRIMSON to be viewed as a more general and reusable evaluation framework, rather than a specific GPT-5.2 + prompting pipeline, it would be helpful to include additional analysis of robustness, such as consistency across different backbone models or at least run-to-run variance. Such experiments would better clarify whether the gains primarily come from the framework design itself or depend substantially on one particular model-and-prompt combination.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I gave this paper a Weak Accept because, overall, I find the contribution meaningful and sufficiently supported, although there are still some limitations that prevent me from rating it higher. On the positive side, the paper addresses an important and clinically well-motivated problem. Existing radiology report evaluation metrics often do not adequately distinguish the clinical consequences of different errors, whereas this work integrates patient context, clinical significance, and fine-grained error modeling into a unified framework. I also find the overall methodological design thoughtful, especially the combination of severity-aware weighting, attribute-level discrepancy analysis, and the treatment of normal findings, which makes the evaluation logic more aligned with real radiologist reasoning. In addition, the experimental validation is fairly strong: beyond correlation analysis on ReXVal, the paper introduces RadJudge and RadPref, which provide complementary evidence that the proposed framework aligns better with expert judgment than prior metrics. I did not rate the paper more highly because some limitations remain. First, the framework is strongly dependent on a powerful closed-source LLM judge, and comparison with other closely related LLM-based evaluation methods is still limited. As a result, it is not yet entirely clear to what extent the gains come from the framework design itself, as opposed to the specific model-and-prompt combination used here. Second, while the paper provides targeted validation for several key design ideas, its robustness analysis is still incomplete, particularly with respect to backbone sensitivity or run-to-run variance. Overall, I think this is a well-motivated paper with a clear framework, meaningful empirical support, and likely community value. Although I would like to see stronger robustness analysis and broader comparison with other LLM-based evaluators, I view these as limitations that are addressable rather than fatal weaknesses, which is why I lean toward Weak Accept.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper introduces CRIMSON, a clinically grounded, LLM-based evaluation framework for radiology report generation. Its key contribution is a severity-aware scoring system that incorporates patient context (e.g., age, indication) and assigns clinically meaningful weights to different types of errors, including missing, false, and attribute-level discrepancies. Unlike prior metrics, CRIMSON prioritizes clinically consequential mistakes and supports partial credit. The authors also provide comprehensive validation against radiologist judgments and introduce new benchmarks (RadJudge, RadPref), demonstrating improved alignment with expert clinical preferences. Additionally, a fine-tuned MedGemma model enables privacy-preserving, local deployment of the evaluation pipeline.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Clinically grounded evaluation paradigm: The paper introduces a severity-aware evaluation framework that aligns scoring with real clinical impact, prioritizing clinically consequential errors (e.g., missed urgent findings) over benign discrepancies. This is a meaningful improvement over prior metrics that treat errors uniformly or rely on surface-level similarity.

    Integration of patient context: Incorporating patient-specific factors such as age and clinical indication into evaluation is a novel and important contribution, better reflecting real-world radiologic reasoning and improving the clinical validity of the metric.

    Fine-grained and interpretable error modeling: The proposed taxonomy of false findings, missing findings, and attribute-level errors (with partial credit) provides a transparent and interpretable evaluation framework that can be useful for both benchmarking and model debugging.

    Strong clinical validation: The method is evaluated against radiologist-annotated error counts and preferences, showing strong correlation and improved alignment compared to prior metrics. The inclusion of multiple expert radiologists strengthens the credibility of the results.

    Introduction of clinically relevant benchmarks: The proposed RadJudge and RadPref benchmarks provide valuable testbeds for future research and highlight clinically meaningful evaluation scenarios beyond standard metrics.

    Practical deployment considerations: The fine-tuned MedGemmaCRIMSON model enables privacy-preserving, local evaluation, which is highly relevant for real-world clinical adoption where data-sharing constraints are critical.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Limited generalizability: The framework is specifically designed for chest X-ray reports, with task-specific severity definitions and attribute rules. Extending CRIMSON to other imaging modalities (e.g., CT, MRI) would likely require substantial redesign of the taxonomy and clinical weighting scheme.

    Dependence on a proprietary LLM evaluator: The core CRIMSON pipeline relies on GPT-5.2 for finding extraction, error classification, and severity assignment. While the MedGemmaCRIMSON model enables local deployment, it is trained to approximate GPT-5.2 outputs. It therefore remains unclear how much of the performance gain is attributable to the proposed scoring framework versus the strength of the underlying LLM evaluator.

    Lack of ablation studies: The paper does not isolate the contribution of key components such as clinical significance weighting, patient context integration, and partial credit for attribute errors. This makes it difficult to assess which design choices are most responsible for the observed performance gains.

    Potential benchmark bias: The RadJudge benchmark is constructed around clinically nuanced scenarios aligned with the design principles of CRIMSON. While useful, this may partially favor the proposed method and does not fully substitute for independent external validation.

    Subjectivity of clinical weighting: The severity weights and error categorization are based on expert-defined guidelines. While clinically motivated, these may vary across institutions and clinical practices, potentially affecting robustness and generalizability.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The main factors supporting this recommendation are the clear methodological contribution and strong clinical validation. The proposed severity-aware, context-sensitive evaluation framework addresses key limitations of prior metrics by aligning scoring with real clinical impact. The paper is well-motivated, technically sound, and introduces meaningful benchmarks (RadJudge, RadPref) with extensive radiologist evaluation, showing strong empirical gains.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper introduces CRIMSON, an LLM-based evaluation metric for chest X-ray report generation. The framework operates in three stages: (1) extracting abnormal findings and assigning four-tier clinical significance weights (urgent / actionable non-urgent / non-actionable / expected benign) informed by patient age and indication, (2) classifying discrepancies into false findings, missing findings, and eight attribute-level error types, and (3) computing a severity-weighted composite score bounded on (−1, 1]. CRIMSON is validated on three benchmarks: correlation with radiologist-annotated significant error counts in ReXVal (n = 50), a 30-case pass–fail clinical judgment suite (RadJudge), and a 100-case pairwise radiologist preference benchmark (RadPref). The paper also releases RadJudge and RadPref as community resources.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Clinically well-motivated severity taxonomy. The four-tier significance framework is well-calibrated to radiological practice. The paper provides concrete, clinically accurate examples — e.g., ETT malpositioning classified as urgent, age-appropriate aortic calcification as expected benign — and integrates established guidelines such as Fleischner Society nodule measurement thresholds (MacMahon et al., Radiology 2017) into the attribute error rules. The explicit separation of non-actionable from expected benign findings, justified by inter-radiologist variability in reporting style, reflects genuine domain understanding.

    2.Patient context sensitivity as a novel contribution. Incorporating patient age and indication into severity assignment — such that aortic calcification in a 25-year-old is weighted differently from the same finding in a 75-year-old — is clinically correct and appears to be a meaningful and relatively underexplored addition to existing evaluation metrics. That said, the paper’s claim of incorporating “full clinical context” appears overstated: in practice, the framework uses age and indication, whereas clinically important contextual factors such as prior imaging, ICU/postoperative status, support devices, and known underlying disease also substantially alter the significance of reporting discrepancies. The authors should better delimit the scope of “context” as implemented.

    3.Multi-benchmark validation strategy. The three complementary validation paradigms (error count correlation, clinical judgment ranking, preference alignment) provide triangulated evidence. Confidence intervals are reported throughout Table 1, which is commendable and rare in metric evaluation papers. The 5-run averaging to account for LLM non-determinism is methodologically appropriate.

    4.RadPref benchmark as a community contribution. RadPref — 100 pairwise cases rated by three cardiothoracic radiologists on a 1–5 scale with structured error categorization — addresses a genuine gap in evaluation infrastructure. The diverse candidate generation regime (MedGemma, random sampling, BERT-matched, LLM editing) ensures coverage of varied error profiles. This is a valuable standalone contribution independent of the metric itself.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Framework contribution not separable from backbone model capability. CRIMSON uses GPT-5.2 as its backbone, while comparison metrics rely on different underlying models. Because CRIMSON’s framework and backbone choice are introduced together, the paper does not provide a backbone-controlled comparison, making it difficult to attribute the observed gains specifically to the proposed clinical design (severity taxonomy, patient context, attribute-level errors) versus the language understanding capability of the backbone. The authors should clarify why they believe the framework design, rather than the backbone, is the primary driver of the reported improvements.

    2.Comparison set excludes recent closely related metrics. The paper references several recent evaluation methods in the related work section — including FineRadScore [4], LLM-RadJudge [15], GEMA-score [22], and ReEvalMed [6] — but limits empirical comparison to older metrics (GREEN, RadGraph, CheXbert, RadCliQ-v1). Given that some of these recent methods also incorporate LLM-based evaluation or severity-aware assessment, their exclusion from the experimental comparison makes it difficult for readers and the AC to assess CRIMSON’s relative contribution against the current state of the art.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    CRIMSON addresses a genuine clinical need with a well-motivated design. The four-tier severity taxonomy with patient context sensitivity is a meaningful and underexplored contribution, RadPref is a valuable community benchmark, and the three-pronged validation strategy with transparent confidence intervals demonstrates methodological care. The paper is clearly written and the code is publicly available.

    Two concerns temper my enthusiasm. The framework’s contribution cannot be cleanly separated from the GPT-5.2 backbone (W1), and the comparison set excludes several recent LLM-based metrics that the paper itself cites (W2). These issues do not invalidate the contribution — the clinical design principles (severity weighting, context sensitivity, attribute-level errors) are substantive and would not emerge from a backbone upgrade alone — but they leave some ambiguity in how much of the quantitative gain is attributable to the framework versus the model. A discussion addressing these points would strengthen the paper.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

N/A



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper presents CRIMSON, an evaluation framework for chest X-ray report generation. The paper contributes a unified severity-aware and context-aware evaluation formulation (patient age/indication plus finding-level clinical weighting), a fine-grained taxonomy with attribute-level errors, and strong validation complemented by RadJudge and RadPref benchmarks demonstrating stronger agreement with expert judgment than prior metrics.

    The paper presents a clinically meaningful and methodologically thoughtful contribution to radiology report evaluation. The work addresses an important gap in existing metrics by moving beyond surface similarity and uniform error counting toward clinically grounded scoring. The strongest concerns are that the gains of the framework are difficult to disentangle from the capability of the proprietary LLM judge (GPT-5.2), and there are limited empirical comparisons to existing metrics and ablations. Nevertheless, the paper’s validation across multiple radiologist-centred benchmarks provides credible evidence that the proposed evaluation direction is useful to the community, and future work should further separate framework vs. backbone effects and broaden comparisons.



back to top