List of Papers Browse by Subject Areas Author List
Abstract
Traditional metrics for Medical Report Generation (MRG) predominantly rely on surface-level n-gram overlap, which fails to capture clinical factual accuracy and often overlooks catastrophic diagnostic errors. We address this fundamental limitation by proposing \textbf{AtomiMed}, a universal, modality-agnostic evaluation framework that decomposes complex medical narratives into a standardized, multi-level hierarchy of Atomic Clinical Facts, encompassing Disease-level entities and Attribute-level descriptors, including location, morphology, and severity. By implementing an Agentic Cross-Verification loop between ground-truth and predicted reports, AtomiMed simulates a multi-radiologist peer-review process to verify clinical consistency, thus enabling the decoupled assessment of diagnostic detection and descriptive accuracy. To facilitate standardized evaluation, we introduce \textbf{MRGEvalKit}, an open-source toolkit for automated hierarchical extraction, and curate \textbf{OmniMRG-Bench}, a comprehensive multi-modal benchmark covering X-ray, CT, MRI, and Ultrasound. Extensive experiments on multiple expert-annotated reader studies demonstrate that AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2351_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Venn2336/MRGEvalkit
Link to the Dataset(s)
N/A
BibTex
@InProceedings{WanYua_AtomiMed_MICCAI2026,
author = { Wang, Yuan AND Chang, Wanxing AND Jiang, Songtao AND Gao, Shujian AND Zhang, Xiaotian AND Yuan, Ruifeng AND Cao, Weiwei AND Shi, Bowen AND Zhang, Ling AND Liu, Zuozhu AND Zhang, Jianpeng},
title = { { AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16882},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution of this paper is the introduction of AtomiMed, a modality-agnostic, clinically grounded evaluation framework for medical report generation that moves beyond traditional surface-level metrics by decomposing reports into a structured hierarchy of Atomic Clinical Facts (ACFs) and verifying their consistency through a bidirectional agentic cross-verification process. Specifically, the framework separates evaluation into disease-level detection and attribute-level description (e. g. , location, severity, morphology), enabling fine-grained and interpretable assessment of clinical correctness. In addition, the paper contributes a large-scale multi-modal benchmark (OmniMRG-Bench) spanning X-ray, CT, MRI, and Ultrasound, along with an open-source toolkit (MRGEvalKit), facilitating standardized and reproducible evaluation. Together, these contributions provide a more reliable, interpretable, and clinically aligned alternative to existing metrics, with demonstrated improvements in correlation with radiologist judgment across multiple datasets.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Good Motivation: The authors address a critical limitation in current Medical Report Generation (MRG) evaluation, as traditional n-gram metrics fail to capture clinical factual accuracy. Proposing a fact-based, modality-agnostic evaluation framework is highly necessary for clinical applications.
- Novel Evaluation Framework: Decomposing complex medical reports into a hierarchical structure (Disease-level and Attribute-level) and utilizing an Agentic Cross-Verification loop is a sophisticated and highly interpretable approach.
- Comprehensive Benchmark: The curation of OmniMRG-Bench is a significant contribution. Covering four imaging modalities (X-ray, CT, MRI, Ultrasound) and providing a large scale of expert-verified Atomic Clinical Facts provides an excellent resource for standardized evaluation.
- Solid Experimental Results: The experiments are well-designed and convincing. AtomiMed demonstrates a significantly higher correlation with human radiologist judgments compared to both traditional metrics and the state-of-the-art LLM metric (GREEN), especially demonstrating strong generalization in non-X-ray modalities.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
This is a good paper but a few things could be done to improve this work:
- Missing Hallucination Analysis: The entire framework heavily relies on the initial atomic decomposition step performed by an LLM. However, LLMs are prone to hallucination or information omission. The authors may provide a rigorous statistical evaluation of the LLM’s accuracy during this specific decomposition phase.
- Questionable String Matching: The method aligns disease entities to attribute-level keys using fuzzy string similarity with a threshold of 0.8.This is potentially risky in the clinical domain, where minor string differences can have entirely opposite meanings. The authors should provide evidence or ablation studies to prove the robustness of this matching strategy over semantic mapping.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The Accept recommendation is primarily driven by the paper’s clear conceptual innovation, strong empirical validation, and high practical relevance to clinical AI evaluation. The introduction of a hierarchical Atomic Clinical Fact (ACF) representation combined with a bidirectional agentic cross-verification mechanism provides a principled and interpretable alternative to existing evaluation metrics, effectively addressing the long-standing issue that n-gram and even semantic metrics fail to capture clinical correctness. Unlike prior approaches, the method explicitly separates disease detection from attribute-level description, enabling fine-grained error attribution that aligns closely with radiologist reasoning. This is further strengthened by the creation of OmniMRG-Bench, a comprehensive multi-modal benchmark, and consistent improvements in correlation with expert human judgment across multiple datasets, as shown in Table 2 (page 6).
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This work introduces AtomiMed, a modality-agnostic framework for evaluating automated medical report generation (MRG) by decomposing reports into a hierarchy of Atomic Clinical Facts (ACFs)—capturing both disease-level findings and their associated attributes such as location and severity—and evaluates them through a bidirectional Agentic Cross-Verification process that separates diagnostic detection from descriptive accuracy. The authors present OmniMRG-Bench, a multi-modal benchmark spanning X-ray, CT, MRI, and ultrasound with expert radiologist annotations, along with MRGEvalKit, an open-source toolkit for reproducible hierarchical scoring.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The proposed evaluation framework is modality-agnostic and largely model-agnostic. While some prior works demonstrate generalization to out-of-domain modalities, the design here is inherently not tied to a specific modality, which improves its broader applicability. 2.The paper includes a thorough empirical evaluation across multiple benchmarks and datasets, complemented by radiologist assessment, which strengthens the validity of the findings.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The description of attribute-level scoring lacks clarity. The authors state that “only findings correctly aligned in both directions contribute attribute questions” and that “attribute precision and recall are computed analogously over the aligned finding set.” This suggests that attribute comparisons are performed only when the corresponding disease is mentioned in both the ground truth and predicted reports. If this is the case, it is unclear why the prostate hyperplasia example appears in the Agentic Cross-Verification recall process in Fig. 1.This inconsistency makes it somewhat unclear how the attribute-level evaluation is implemented in practice, and whether it may have an impact on the reported results. Additionally, the authors may want to verify the recall calculation shown in Fig. 1, as it appears that the value should be 1/4.2.The experimental setup for baseline comparisons is not described with sufficient clarity. It is unclear whether the authors report official scores from prior work or recompute the metrics on the same datasets for the evaluations in Table 2.For instance, some of the reported correlation metrics appear inconsistent with prior literature. For instance, the Kendall’s Tau for GREEN on ReXVal is reported as 0.626, which is lower than the value reported in the original paper (0.64). Similarly, RaTEScore on RaTE-Eval shows substantially higher Kendall’s Tau (0.462) and Spearman’s rho (0.608) compared to the values reported in this work (0.339 and 0.46, respectively). Clarifying these discrepancies would help ensure a fair and reproducible comparison.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
1.The description of the annotation process for OmniMRG-Bench could be made more precise. The paper states that “ACFs are extracted from ground-truth reports and verified by board-certified radiologists to ensure annotation fidelity,” and reports a total of over 178K QA pairs. It is later mentioned that radiologists reviewed 80 sampled cases for the Human Preference Gold Standard procedure. As written, this could give the impression that all 178K QA pairs were directly verified by radiologists. If this is not the case, the authors may consider revising the wording to more accurately reflect the extent of expert involvement. 2.Additional details would improve clarity regarding the Human Preference Gold Standard described in the Implementation and Evaluation Protocol. For example, the paper states that “a board-certified radiologist independently reviewed the 10 models’ outputs for all 80 sampled cases,” but it is unclear how these 80 cases are distributed across modalities, how the evaluation was conducted, and what criteria were used for comparison. Providing these details would strengthen the transparency and reproducibility of the study.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend acceptance as the paper presents a simple but well-motivated and potentially impactful approach to evaluating medical report generation. The proposed framework introduces a structured, modality-agnostic evaluation based on Atomic Clinical Facts. The OmniMRG-Bench and MRGEvalKit, along with evaluation across multiple datasets and radiologist assessment, further strengthens the contribution.
While there are some clarity issues regarding attribute-level scoring, baseline comparisons, and human evaluation details, these appear to be addressable. Overall, the strengths and practical value of the work outweigh these concerns.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes AtomiMed, a hierarchical evaluation framework for medical report generation. It decomposes reports into disease-level and attribute-level atomic clinical facts, and then applies bidirectional cross-verification to assess disease detection and attribute description separately. The paper also introduces OmniMRG-Bench, a multi-modal benchmark covering X-ray, CT, MRI, and ultrasound, together with MRGEvalKit for automated evaluation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- It addresses an important problem. The paper focuses on a real weakness of current medical report evaluation, namely the gap between surface-level text similarity and true clinical correctness.
2.The formulation is clinically meaningful and interpretable.
- The empirical study is relatively broad.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The methodological novelty is limited. The idea of converting reports into structured clinical facts before comparison is not new. Prior work on radiology information extraction, graph-based evaluation, and LLM-based evaluation already follows a similar direction. The hierarchical QA design and bidirectional verification are reasonable extensions, but they do not yet seem clearly distinct enough from existing approaches.
2.The method relies on a strong backbone model for both decomposition and verification, but the paper provides limited analysis of cost, robustness, reproducibility, or sensitivity to model choice. This makes it difficult to tell whether the gains come from the proposed framework itself or from the capability of the underlying model.
3.Although the paper claims expert verification, it provides limited detail on annotation protocol, agreement, and quality control. This weakens confidence in the benchmark as a strong community resource.
4.The state-of-the-art claim is not justified by the current evidence. The method does not consistently outperform prior approaches across all benchmarks, and the supporting human evaluation remains limited in scale. More broadly, the paper is better understood as a benchmark and evaluation pipeline paper than as a true methodology paper, with its main contribution lying in resource curation and the packaging of existing ideas rather than in a clearly new technical advance.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the paper addresses an important problem and presents a reasonably complete empirical study, the core contribution appears too incremental for acceptance in its current form. In addition, the framework depends heavily on a very large language model, while robustness, ablation, and benchmark validation remain limited.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
N/A
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Paper Summary: This paper proposes a modality-agnostic, clinically grounded evaluation framework for medical report generation. The method is evaluated on multiple datasets, with reported improvements in correlation with expert human judgment.
Strengths: The reviewers (R #1 and R #2) identified several strengths, including clear conceptual innovation, strong empirical validation, and high practical relevance to clinical AI evaluation. I broadly agree that the paper’s main contribution is the introduction of a hierarchical Atomic Clinical Fact (ACF) representation combined with a bidirectional agentic cross-verification mechanism. This idea provides a principled and interpretable evaluation metric and helps better capture the clinical value of the proposed method.
Weaknesses: The main concerns raised by the reviewers include clarity issues regarding attribute-level scoring, baseline comparisons, and details of the human evaluation, as well as the lack of a rigorous statistical evaluation of the LLM’s accuracy.
Review Reconciliation: Reviewer opinions are largely consistent, particularly between R #1 and R #2.After considering the reviews and the manuscript, I agree with R #1 and R #2 regarding the paper’s contribution and technical novelty.
Decision: The paper is technically sound, addresses an important problem in medical report generation, and presents a meaningful contribution beyond existing work. The empirical validation is convincing, and no major technical weaknesses were identified that would materially affect acceptance. I therefore recommend Early Accept.