Abstract

The chest and abdomen are critical, anatomically complex regions with diverse pathologies, with digital radiography (DR) serving as the primary modality for screening and diagnosis. Developing an effective DR foundation model with strong disease discrimination through self-supervised learning remains a significant challenge. Moreover, prior research has largely overlooked abdominal DR and the potential of computed tomography (CT) localizers as a bridge to fine-grained CT knowledge. In this work, we pioneer the incorporation of large-scale, previously neglected CT localizers and CT reports to train a unified DR foundation model on both chest and abdominal regions. We propose a dual-branch contrastive pretraining approach using 1.1 million DR and 1 million CT localizers, each paired with radiologist reports. By integrating comprehensive knowledge from CT reports, we substantially enhance the model’s capability in DR interpretation. Extensive evaluations on 10 downstream tasks demonstrate the superiority of our model. To the best of our knowledge, this is the first foundational pretraining framework that jointly leverages DR, CT localizers, and reports, scaling to an unprecedented 2.1 million image-report pairs. It offers valuable insights into CT-enhanced intelligent DR diagnosis and scalable multimodal medical foundation models, with demonstrated robustness across diverse clinical tasks. The model is available at https://github.com/sunyh1/DEER.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1156_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/sunyh1/DEER

Link to the Dataset(s)

https://arxiv.org/abs/2405.19538 https://arxiv.org/abs/2505.00228 https://doi.org/10.17632/4gvcb6gmh2.1 https://doi.org/10.3390/app13158646 https://doi.org/10.1038/s41597-023-02460-0 https://doi.org/10.3389/fcvm.2024.1351965 https://doi.org/10.1109/JBHI.2020.3037127 https://doi.org/10.17632/wndbd5r26y.2 https://doi.org/10.5281/zenodo.5548363 https://doi.org/10.1038/s41467-021-21311-3 https://doi.org/10.1007/s10278-022-00706-8 https://arxiv.org/abs/2408.02900

BibTex

@InProceedings{SunYih_DEER_MICCAI2026,
        author = { Sun, Yihua AND Guo, Jia AND Wang, Qijing AND Ouyang, Jingzhou AND Li, Ge AND Zhang, Nan AND Chen, Fang AND Liao, Hongen},
        title = { { DEER: A Foundation Model with Dual-Branch CT-Enhanced Embeddings for Thoracoabdominal Radiographs } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16878},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes DEER, a foundation model for thoracoabdominal digital radiograph (DR) interpretation. The key idea is to use CT localizers as a bridge to incorporate richer diagnostic information from CT reports into DR representation learning. The authors design a dual-branch contrastive learning framework where one branch aligns DR images with DR reports and another aligns CT localizers with CT reports, sharing a ViT backbone. The model is pretrained on 2.1 million image-report pairs (1.1M DR + 1M CT localizers) and evaluated on 10 downstream classification tasks spanning chest, abdomen, pelvis, and spine.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The idea of leveraging CT localizers as a previously overlooked data source is novel and clinically well-motivated. Localizers are routinely acquired during CT scanning but rarely utilized for model training. Pairing them with diagnostically rich CT reports to enhance DR representations is a creative use of existing clinical data. 2.The scope of evaluation is broad, covering 10 downstream tasks across multiple anatomical regions (chest, abdomen, pelvis, spine) and diverse pathologies (pulmonary diseases, congenital heart diseases, COVID-19, fractures, developmental dysplasia, spondylolisthesis).

    1. The paper addresses the underexplored area of abdominal DR interpretation, which has practical clinical value. Most existing DR foundation models focus exclusively on chest X-rays. 4.The paper is generally well-written and the figures (especially Fig. 1) effectively communicate the overall framework and data composition.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The dual-branch design is essentially standard CLIP contrastive learning with two separate CLS tokens and projection heads sharing a ViT backbone. The contribution is primarily data-driven (scale and data source) rather than method-driven.

    2.The ablation in Fig. 3e only compares “with vs. without localizer,” but adding localizers also adds 1 million extra training samples.

    3.Since DR and CT localizer images are not paired (they come from different patients and examinations), the transfer of CT diagnostic knowledge to DR representations relies entirely on implicit sharing through the backbone weights. However, no analysis is provided to verify that this transfer actually occurs. The authors could provide: (a) probing experiments testing whether DR-branch features can predict CT-report entities, (b) attention map comparisons across branches, or (c) feature similarity analysis between the two branches. Without such evidence, the “CT-enhanced” claim remains unsubstantiated.

    4.All 10 tasks are classification tasks. For a foundation model built on a CLIP framework, the absence of zero-shot classification and linear probing experiments is a significant gap. These are the standard protocols for evaluating the quality of pretrained representations and would more directly reflect the benefit of pretraining than fine-tuning, which can mask representation quality differences.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents an interesting idea, using CT localizers to enhance DR representations. However, I have several concerns as detailed in the weaknesses section, most notably the limited methodological novelty, the failure to disentangle localizer-specific contributions from data scale effects and the lack of evidence for the claimed cross-branch knowledge transfer. 

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The manuscript proposes DEER, a foundation model for digital radiography image understanding. DEER adopts a dual-branch cross-modal pretraining strategy, incorporating CT localizers and their corresponding textual reports to enhance DR image representation. The model is pretrained on the largest dataset to date comprising 2.1 million image–report pairs, and its effectiveness is validated across a wide range of downstream tasks covering diverse anatomical regions.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The manuscript leverages CT localizers as a bridge to incorporate the richer information contained in 3D CT reports into the training process, thereby facilitating the understanding of 2D digital radiography images. The resulting dataset constructed under this paradigm is valuable in both modality diversity and scale. 2.A shared encoder is used for DR and CT localizer images, coupled with separate projectors to refine their respective representations, constitutes a reasonable and meaningful dual-branch design. 3.The downstream experiments and evaluation protocols are comprehensive. 4.The manuscript is well-written and generally easy to follow.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The motivation of the manuscript appears to stem from two aspects: (1) the underexplored potential of CT localizers, and (2) the neglect of abdominal DR. In addition, the authors state in the abstract that “Developing an effective DR foundation model with strong disease discrimination through self-supervised learning remains a significant challenge.” While these claims are reasonable, they would benefit from a more detailed discussion and stronger grounding in related work. In particular, it would be helpful to specify which existing models have adopted what types of pretraining strategies, as well as the modality and scale of the datasets used, so that these high-level assertions can be better contextualized and more convincingly supported. 2.The experimental implementation details require more thorough description. For instance, it is unclear whether all models are evaluated under a unified experimental setting or with their individually optimized configurations (e.g., input image resolution), and which datasets lack predefined train–test splits and therefore require cross-validation. 3.The statistical significance analysis presented in this work requires further clarification. For example, how the comparison pairs for the t-test are defined and selected. 4.It can be observed that some downstream tasks exhibit relatively large standard deviations (e.g., COVIDGR and PelviXNet), which in turn lead to higher p-values in the t-test. The underlying reasons for this phenomenon should be further analyzed and explained. 5.Section 3.5 can be regarded as an ablation study evaluating the effect of incorporating LOC into the pretraining process. However, the experimental setup requires more detailed description. In particular, a more rigorous design would control for the scale of pretraining data to ensure that the observed performance gains arise from the additional modality rather than simply from increased data scale. In addition, as I understand that the y-axis in Fig. 3(e) represents percentages, which should be explicitly labeled or clarified to avoid potential confusion.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The manuscript presents a clear motivation, with a well-designed and effective methodology. The downstream experiments cover a broad range of tasks, and the evaluation is comprehensive, with results presented in a clear and objective manner. The paper is well-structured, logically organized, and easy to follow. In addition, the authors’ commitment to releasing the proposed foundation model would constitute a valuable contribution to the community. Although certain aspects require more detailed explanation, likely due to space limitations, the overall coherence and completeness of the pipeline meet the standard for acceptance.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The concept of incorporating CT localizers and their corresponding textual reports to enhance DR image representation in this paper is novel, and the proposed method is well-designed and effective.



Review #3

  • Please describe the contribution of the paper

    The paper’s main contribution is the introduction of DEER, a large-scale thoracoabdominal radiograph foundation model that uses dual-branch contrastive pretraining with DR, CT localizers, and reports to learn more disease-discriminative radiograph representations.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The problem setting is important and clinically meaningful. The paper goes beyond chest-only radiograph foundation models and expands the scope to thoracoabdominal radiographs, with additional transfer evaluation on pelvic and spinal tasks. 2.The use of CT localizers as a bridge is an interesting idea. Leveraging CT localizers and CT reports to enhance DR representations is a novel and intuitively motivated design choice. 3.The scale of pretraining and the breadth of evaluation are strong. The model is pretrained on 2.1 million image-report pairs and evaluated across 10 downstream tasks covering chest, abdomen, pelvis, and lumbar spine. 4.The empirical results are strong. DEER shows competitive or superior performance across multiple datasets, and the paper presents consistent gains from image-report pretraining and additional LOC incorporation.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The source of the performance gain is not sufficiently disentangled. Although the paper emphasizes the architectural advantages of DEER, the proposed method also benefits from substantially larger-scale data and strong pretrained initialization, making it unclear how much of the improvement is truly attributable to the CT-enhanced dual-branch design itself. As a result, the current results do not fully isolate the methodological contribution of the proposed framework. 2.The paper does not fully establish whether LOC and DR are sufficiently distinct to justify separate branch-specific modeling. Since both are fundamentally 2D projection-based radiographic images, it remains unclear whether the observed gains from incorporating LOC reflect true modality complementarity or are instead driven by confounding factors such as increased data scale, richer CT-report supervision, or broader anatomical coverage. 3.A further concern is the potential misalignment between CT localizers and CT reports. Because CT localizers are typically acquired before the actual CT scan for scan planning and field-of-view determination, the final CT coverage may differ from that of the localizer. Consequently, some findings described in the CT report may correspond to regions that are only partially covered, or not visible at all, in the localizer image, which could weaken the assumption that localizers provide well-aligned supervision for report-guided representation learning.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please refer to the weaknesses.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I have carefully read the authors’ rebuttal and appreciate their effort in providing detailed and thoughtful responses. The rebuttal clarified several aspects that were previously unclear to me. While some concerns remain, to the best of my knowledge, this is the first work to leverage CT line-of-consistency (LOC) in this context. I believe that the proposed formulation may serve as a useful foundation for future studies building on this direction. Therefore, I am willing to raise my score to the acceptable range.



Author Feedback

We sincerely thank the reviewers for their valuable suggestions and for recognizing the novelty of using LOC and the clinical significance toward thoracoabdominal DR. The final version will be revised accordingly and will include the following information.

Q(R1): Contribution is data-driven rather than method-driven. A: LOC are visually similar to DR while naturally paired with CT reports, yet they have been largely overlooked in prior research. Our novelty and methodological contribution are identifying LOC as an effective bridge for enhancing DR learning, which may provide useful insights for future foundation model development.

Q(R1, R2, R3): In Fig. 3e, is the improvement brought by LOC due to additional 1M data samples, broader anatomical coverage, or the modality itself? A: During this work, we conduct exploratory experiments. We did not observe consistent improvements in DR tasks when adding modalities other than LOC. Specifically, we sampled 1M non-DR image-text pairs from MedTrinity-25M, including modalities such as CT and MR across anatomical regions like chest and abdomen. When adding these additional 1M samples to the DR training data, performance decreases (CHD-CXR -0.5%, PelviXNet -0.9%, compared to DEER). On BUU-LSPINE, it decreases -12.4% from DEER, even substantially worse than training without additional data. In contrast, using LOC and CT reports led to improvements, suggesting that the gain mainly comes from LOC and CT reports rather than simply from increasing the data scale. LOC can serve as an additional modality, further boosting DR foundation models beyond large-scale DR datasets.

Q(R1): Zero-shot and linear-probing. A: We did not include zero-shot evaluation because most of the compared methods are not directly applicable in a zero-shot setting. Since baseline models are not pretrained on abdominal data, we focus on the fine-tuning setting to ensure a fair comparison under MICCAI’s page limit. In fact, as a CLIP framework, DEER has advantages under linear-probing (CHD-CXR +1.4%, PelviXNet +0.6%, over the best baseline).

Q(R3): Justify separate branch-specific modeling. A: Even though both DR and LOC are 2D projection images, LOC has a lower dose and poorer image quality compared to DR (Fig. 1), resulting in a domain gap. Before full evaluation, we conducted ablation studies on a subset of the data. The results show that directly mixing DR and LOC for joint training without a dual-branch design leads to degraded performance (CHD-CXR -0.5%, PelviXNet -1.2%).

Q(R3): Pretrained initialization. A: Using pretrained initialization is a standard practice. RAD-DINO & MedGemma are both initialized from large-scale pretrained DINOv2 & Gemma. DEER still outperforms.

Q(R3): LOC and CT reports may not align well. A: LOC are used for CT scan planning, and in most cases their coverage is broader than the final CT, thereby encompassing anatomy described in the CT report. Even providing only weak supervision, our novelty is introducing largely overlooked LOC into DR model training and boosting performance, as in “Q(R1, R2, R3)”.

Q(R1): Further evidence for knowledge transfer from CT to DR. A: We appreciate this insightful suggestion. While DEER potentially enables predicting CT report entities, evaluating this would require a DR-CT paired dataset, which is very rare and beyond the scope of this work. Due to page limits, data source limits, and restrictions on supplementary material, we focus the main paper on primary results and key insights. In the future we will explore CT entity prediction (Sec 4), and detailed feature-level analyses.

Q(R2): Implementation details. A: We use each model’s native pretraining resolution and apply unified settings for fine-tuning. The t-test is performed across runs of different models. For datasets without an official test split, we adopt 5-fold cross-validation and report the mean and std across folds (Sec 3.1). The relatively large std is due to inter-fold variability.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    reviewers agree the paper is well motivated, and the main concerns are methodological rather than presentational, reviewers’ concerns should be addressed in rebuttal before acceptance

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    the reviewers acknowledges the idea is novel, clinically meaningful, and empirically promising, the authors have also addressed reviewers’ concerns well with evidence.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The authors have addressed most of the concerns raised by reviewers. I recommend Accept.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper leverages CT localizers and CT reports to enhance thoracoabdominal DR foundation-model pretraining. Although the reviewers raised valid concerns about whether the gains come from LOC-specific knowledge transfer versus added data scale, the rebuttal provided useful clarifications and additional evidence that LOC contributes more effectively than simply adding other modalities or more data. Overall, the work is well motivated and broadly evaluated.



back to top