List of Papers Browse by Subject Areas Author List
Abstract
Radiology reports for the same patient examination may contain clinically meaningful discrepancies arising from interpretation differences, reporting variability, or evolving assessments. Systematic analysis of such discrepancies is important for quality assurance, clinical decision support, and multimodal model development, yet remains limited by the lack of standardized benchmarks. We present RADAR, a multimodal benchmark for radiology report discrepancy analysis that pairs 3D medical images with a preliminary report and corresponding candidate edits for the same study. The dataset reflects a standard clinical workflow in which trainee radiologists author preliminary reports that are subsequently reviewed and revised by attending radiologists. RADAR defines a structured discrepancy assessment task requiring models to evaluate proposed edits by determining image-level agreement, assessing clinical severity, and classifying edit type (correction, addition, or clarification). In contrast to prior work emphasizing binary error detection or comparison against fully independent reference reports, RADAR targets fine-grained clinical reasoning and image–text alignment at the report review stage. The benchmark consists of expert-annotated abdominal CT examinations and is accompanied by standardized evaluation protocols to support systematic comparison of multimodal models. RADAR provides a clinically grounded testbed for evaluating multimodal systems as reviewers of radiology report edits.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5740_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/GeraldSun/RADAR
Link to the Dataset(s)
N/A
BibTex
@InProceedings{SunZha_RADAR_MICCAI2026,
author = { Sun, Zhaoyi AND Jagtiani, Minal AND Yim, Wen-wai AND Xia, Fei AND Gunn, Martin AND Yetisgen, Meliha AND Ben Abacha, Asma},
title = { { RADAR: A Multimodal Benchmark for 3D Image-Based Radiology Report Review } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces RADAR, a multimodal benchmark for image-grounded radiology report revision. The task is formulated to evaluate whether candidate report edits are supported by 3D CT images, along with assessing their clinical severity and categorizing edit types. The dataset is constructed from real-world preliminary-to-attending report revisions, aiming to reflect realistic clinical workflows.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper targets radiology report discrepancy analysis in a realistic workflow (resident-to-attending revision), which is highly relevant to clinical quality assurance and patient safety. This problem setting goes beyond traditional report generation or error detection tasks.
- The benchmark provides useful empirical insights into the limitations of current multimodal models, particularly in image-grounded reasoning and clinically nuanced decision-making (e. g. , severity assessment), which is valuable for the community.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The dataset is relatively small and restricted to a single institution and a single anatomical region (abdominal CT), which raises concerns about statistical robustness and generalizability.
- The dataset is not publicly released and requires a data usage agreement. While understandable due to clinical constraints, the lack of a clear and accessible release plan may limit reproducibility and broader community adoption of the benchmark.
- Despite the strong clinical motivation, the evaluation is limited to comparisons among foundation models. There is no comparison with radiologists, existing clinical QA workflows, or prior methods, making it difficult to assess real-world clinical impact.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
The paper presents a promising direction toward discrepancy-aware evaluation in radiology. To strengthen the work, the authors are encouraged to (1) expand the dataset across institutions and anatomical regions, (2) improve dataset accessibility (e. g. , partial release, benchmark server, or evaluation API), and (3) include comparisons with human experts or existing clinical practices to better demonstrate practical impact.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This score is based on the above strengths and weaknesses.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Although I initially had concerns about the limited dataset scale and accessibility, the rebuttal convincingly clarifies that the main contribution is a clinically meaningful task formulation and an initial benchmark for image-grounded verification of report revisions. The authors also commit to releasing the benchmark through a community challenge platform under a DUA, which improves my confidence in the work’s value and reproducibility.
Review #2
- Please describe the contribution of the paper
The main contribution is RADAR, a benchmark for evaluating whether candidate radiology report edits are justified by 3D CT evidence in a real report review workflow. By combining agreement detection, severity assessment, and edit type classification, the benchmark captures a more realistic and clinically relevant form of multimodal reasoning than prior setups based mainly on binary error detection or report comparison.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper’s strongest point is the task design. It frames radiology report review as judging whether proposed edits are supported by the scan, how important they are, and what kind of change they make. This is more clinically meaningful than simpler benchmark tasks.
The benchmark is also grounded in a real review workflow, using preliminary reports that are later revised by attending radiologists. This makes the setup more realistic and gives the benchmark more credibility. The evaluation is also useful in practice because it reports runtime and compares different input settings.
Finally, the results clearly show that the task is hard for current models. That is a strength of the benchmark, because it shows that the problem is meaningful and not already solved.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Benchmark scale: The dataset size is the main concern. For a paper whose main contribution is a benchmark, the scale should support reliable evaluation. Here, the benchmark is still too small for that standard. This is especially clear in the Natural test set, which contains only five real disagree cases, making disagreement detection on real examples statistically hard to interpret. The results section also misses important supporting analyses, such as stronger significance testing and more detailed breakdowns.
Annotation validity: All annotations come from a single radiologist, and the paper does not report inter-rater agreement for any of the three labels. That makes it difficult to judge how stable or reproducible the benchmark labels are, especially for more subjective dimensions such as severity.
Synthetic data reliance: The benchmark depends heavily on synthetic disagree edits to balance the evaluation set. This design choice is understandable, but its effect on benchmark validity should be discussed more clearly. In particular, the paper should better explain why one model was used for generating synthetic edits and another for evaluation, and what risks this creates for bias or mismatch.
Pilot study scope: Because the benchmark is small, based on a single institution, the current version feels closer to a pilot study than a fully established benchmark. This narrow scope is not necessarily a flaw for an early resource, but it should be discussed more directly in the limitations, especially when considering how well the setup may transfer to other modalities, body regions, or reporting workflows.
Missing analysis: The paper also lacks analysis of what makes cases difficult or what separates model success from failure. More error analysis would make the benchmark much more informative. It would also help to first test how well the models understand the CT images themselves before evaluating the full discrepancy task.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
Minor suggestions: 1) I would bring to the title that it is about discrepancies assessment
2) Use hyperlinks for your references, that if I click on [1] I get to [1]
3) Not sure if both links are needed under [16]
4) Consider using et al for some papers with a million authors. But be consistent.
5) PDF rendering: “meaningful testbed” goes over the line
6) Instead of having Discussion and Conclusion, maybe combine and instead have limitations separate, this would make it easier to distinguish
7) In table 2, just for readability, I would maybe separate Mixed and Natural a bit more or shade the Comp column aggregated view or the main row of the four models you think is most important to compare. Now it takes a bit to understand what I exactly should compare.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important and still underexplored problem: checking whether suggested edits to radiology reports are actually supported by the imaging data. Its strongest contribution is the task itself, and the three-part annotation setup is well thought out. If this is expanded and validated more carefully, it could become a useful resource for the community.
At the same time, the current study feels too limited to fully support the benchmark claim. The dataset is very small, uses only one annotator, and does not report inter-rater agreement. The evaluation of disagreement detection also relies mostly on synthetic examples instead of real clinical cases. There is also no error analysis, no confidence intervals, and no subgroup breakdown.
Overall, I think the paper has a strong and worthwhile idea, but the evidence is not yet strong enough for this to work as a benchmark paper in its current form. Right now, it reads more like a promising pilot study than a fully established benchmark. The paper would be much stronger if the authors could show agreement between annotators, clarify why some important baselines were not included, and report statistical uncertainty more clearly.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper presents RADAR, a novel multimodal benchmark for evaluating image-grounded verification of radiology report edits, based on real clinical revisions and designed to assess fine-grained discrepancy reasoning.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The proposed benchmark is well grounded in a realistic clinical workflow.
- The paper introduces a novel and well-motivated task formulation, a new multimodal task of image-grounded verification of report edits, moving beyond prior work on report generation or synthetic error detection.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The dataset is relatively small (50 CT studies), which raises concerns about statistical robustness and limits the reliability of the empirical comparisons as well as the generalizability of the findings.
- The task assumes the availability of candidate edits and focuses only on their verification, bypassing the more challenging and clinically relevant step of discrepancy detection. This assumption limits the applicability of the approach to real-world deployment scenarios.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- To further strengthen the impact, it would be helpful to expand the dataset in scale and diversity, e.g., including more institutions, imaging modalities, or anatomical regions, to better support generalizability.
- The current baselines are relatively limited. It would be valuable to explore stronger or task-adapted approaches, and more importantly, to consider integrating discrepancy detection and verification into a unified framework, which would better reflect real-world clinical settings.
- It would improve clarity to provide representative examples illustrating each label category (agreement, severity, edit type), helping readers better understand the annotation criteria and task definition.
- Since the annotations are performed by experts, including a human baseline or inter-annotator agreement would help contextualize model performance and provide a clearer sense of the intrinsic difficulty of the task.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Despite the relatively small dataset size, the paper proposes a clinically meaningful and novel task formulation for image-grounded radiology report review. The benchmark is well-motivated and offers a valuable direction for evaluating multimodal models.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
While the benchmark still has limitations in scale and completeness, the paper presents a novel and clinically meaningful problem setting for multimodal radiology reasoning. The proposed evaluation paradigm is realistic and valuable for future research directions. Overall, I view the paper as a borderline accept / weak accept.
Author Feedback
We thank the reviewers and Area Chair for the thoughtful feedback and encouraging assessment of RADAR’s clinical relevance and novelty. We are pleased that all reviewers recognized the importance of the task formulation and its grounding in a realistic radiology workflow. Below we address the major concerns.
1.Benchmark scale, single institution, and “pilot study” framing (R1/R2/R3). We agree that the current benchmark is limited in scale and diversity. Our goal was not to claim that RADAR provides broad benchmark coverage, but to introduce a new multimodal task formulation: image-grounded verification of report revisions derived from real preliminary-to-attending workflows. We will clarify this positioning and strengthen the limitations discussion regarding single-institution abdominal CT data and limited natural disagreement cases. The core contribution is the task setting itself. Existing multimodal radiology benchmarks mainly focus on report generation or synthetic error detection, whereas RADAR introduces a clinically grounded discrepancy-review setting. We believe this distinction supports the work’s value beyond dataset scale, and we will make this framing clearer in the revision.
2.Annotation validity and lack of inter-rater agreement (R2/R3). We appreciate the suggestion regarding additional expert validation and inter-rater agreement analysis, and agree that these additions would strengthen the benchmark. The annotations were performed by an experienced abdominal radiologist using structured guidelines designed to reflect the clinical review workflow. In the revision, we will clarify the annotation protocol in greater detail and include inter-annotator agreement analysis to better quantify annotation consistency. We also recognize that severity assessment contains inherent subjectivity and will discuss this more in limitation.
3.Reliance on synthetic disagreement edits (R2). The scarcity of naturally occurring incorrect edits reflects the clinical workflow: most attending revisions are correct refinements or clarifications, while incorrect edits are rare and often limited to unnecessary wording changes. We introduced synthetic disagreement edits to create a more balanced evaluation setting for image-grounded verification. To maintain transparency, we report “Mixed” and “Natural” evaluations separately. We will further clarify this rationale and discuss the limitations of synthetic augmentation. The “Natural” dataset remains fully composed of real clinical revisions and is intended to reflect real-world performance.
4.Missing baselines, human comparison, and additional analyses (R1/R2/R3). This paper is primarily a benchmark and task definition contribution rather than a system development paper. Accordingly, we evaluated representative multimodal foundation models under consistent input conditions to establish baseline performance on the proposed task. We agree that radiologist-level comparison, subgroup/error analyses, and unified detection + verification pipelines would be valuable, and we position them as future work. The task is designed to isolate image-grounded verification as a foundational subproblem. This is an intentional scoping decision: before addressing full end-to-end discrepancy discovery, it is important to establish whether current multimodal models can determine whether a given report revision is supported by the image. The end-to-end setting is harder because candidate edits are not provided and the model must identify discrepancies without guidance. Demonstrating model limitations in this constrained setting already provides meaningful evidence of the challenges ahead.
5.Reproducibility and accessibility (R1/R2/R3). The benchmark is being released through an established community challenge platform and will be made publicly available upon approval of a Data Use Agreement (DUA). Further details about the data and code will be made available upon publication.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes methodology to assess agreement between an abdominal CT scan and its radiology report and suggest edits. Weaknesses (in evaluation) include a small dataset of 50 abdominal CT studies from a single institution and annotations for training by a single radiologist.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The authors have satisfactorily addressed key concerns raised in the reviews.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The reviewers unanimously agree this work passes the bar for acceptance and presentation at the conference.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The authors provided a valid rebuttal and the major concerns seem to be all addressed. The dataset is a bit small though, looking forward to seeing the extension of the benchmark at a better publication platform. Good luck.