List of Papers Browse by Subject Areas Author List
Abstract
Recent 3D CT vision–language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision–language model on 98k report–volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision–text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., “series X, image Y”), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet–slice pairs and introduce the task of intra-scan snippet localization—predicting the axial depth referred to by a text snippet—reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3986_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/lmb-freiburg/radfinder
Link to the Dataset(s)
N/A
BibTex
@InProceedings{GinSim_Learning_MICCAI2026,
author = { Ging, Simon AND Arnold, Philipp AND Walter, Sebastian AND Alnahas, Hani AND Bast, Hannah AND Kotter, Elmar AND Yang, Jiancheng AND Bozorgtabar, Behzad AND Brox, Thomas},
title = { { Learning to Read Where to Look: Disease-Aware Vision–Language Pretraining for 3D CT } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper builds a new CT foundation model through modifications during contrastive pretraining. Two new losses are added: prompt-based disease supervision and a localization objective. The localisation is enabled by mining 262k such snippets from the internal text reports. These additions enabled an overall better representation learnt, as indicated with improvements over baselines on retrieval tasks, disease classification and the newly introduced localization task.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper targets an important gap in 3D CT vision-language pretraining: most prior research uses only global report-volume alignment, whereas this paper tries to recover finer-grained supervision from the already available report Mining 262k snippet-slice pairs from existing reports is the most interesting part of the paper. Without it it would come across as a “train on more data,” paper. It has a strong empirical story and results that are coherent. The ablations also support the main claims, especially that localization can be added without hurting retrieval/classification much. It is practically interesting that retrieval, disease prompting, and localization are all trained in a shared embedding space instead of bolting on separate heads for each task. That makes the method feel clean and conceptually consistent.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The core advantage may come largely from the internal RefCT dataset size and distribution which, as far as I can tell, will not be released. The new task is interesting, but all localization evidence appears to be on the authors’ own RefCT test set. The classification story is less strong than the retrieval story. The paper claims competitive disease classification, which is fair, but not a clear win. On CT-RATE it matches the best reported AUC rather than surpassing it, and on Rad-ChestCT it is slightly below the best baseline within confidence bounds. The model is also receives much stronger supervision than prior work for this task (although I think Merlin also does something similar with ICD10 codes). It might be good to add a fully supervised baseline for reference.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper’s main strength is the idea of exploiting report-referenced slices as scalable local supervision (an interesting use of weak, local supervision) for 3D CT VLMs, backed by solid retrieval and localization results. Its main weakness is that the most novel part is demonstrated primarily on private, institution-specific data, so external validity is still somewhat uncertain.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
- The authors extracted large-scale 3D report–volume dataset (98k pairs, 50k patients) from which they extracted 262k snippet–slice pairs
- They created a new task of intra-scan snippet localization and showed that it can be trained jointly with global objectives without degrading performances
- The authors trained a 3D CT VLM by combining this new task with contrastive pretraining.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper introduced a novel and well-motivated localization task.
- The ablation clearly shows that the new loss does not degrade performance.
- The scale and quality of the dataset represent a real amount of work.
- The performances of the models are better than every open model available.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Performance gains seem predominantly dataset-driven rather than method-driven. The ablation RefCT only already achieves R@10 26.3, surpassing all published baselines before adding any of the three training novelties. The full model reaches 31.5, but the delta attributable to method vs. data is modest. The paper does not sufficiently address whether a simpler model (e.g., vanilla SigLIP fine-tuning) on the same 98k pairs would perform comparably.
- SPECTRE is trained on A+B+C (no INSPECT); RadFinder adds INSPECT (D) and initializes from SPECTRE’s weights. This compounding advantage makes the reported delta vs. SPECTRE (18.2→31.5) difficult to interpret. A controlled comparison same data, RadFinder method vs. SPECTRE method is missing.
- Some key other competitors could have been added, such as fvlm, TotalFM, or Pillar0.
- The implementation reports an effective batch size of 4096 on a single H100 (96 GB), which necessarily implies gradient accumulation with a small per-step micro-batch. This is not discussed. For SigLIP-style contrastive training, the equivalence between gradient accumulation and a true large batch depends on the implementation of the all-pairs sigmoid loss. The actual number of negatives seen per gradient step, and whether the implementation correctly handles this, should be clarified for reproducibility.
- The localization loss is a strict null result on all global metrics; it neither helps nor hurts, making the “unified model” claim a reframing of a negative result. Moreover, the prompt loss, while clearly beneficial for disease classification, slightly but consistently degrades retrieval on the Merlin benchmark (R@1find 70.7→69.0, R@1impr 42.2→40.3). This trade-off is not discussed.
- The prompt-based disease supervision is never compared against a standard classification head trained on the same binary labels. The added complexity of the prompt approach: positive/negative prompt construction, per-class imbalance weighting, multi-prompt averaging at inference, is only justified if it outperforms a simpler linear probe or MLP head on the same encoder. Without this baseline, it is unclear whether the AUC gains in Table 1 stem from the prompt-in-embedding-space design or simply from the additional disease supervision signal, regardless of how it is parameterized.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
RadFinder makes two genuine contributions that justify acceptance: a large-scale proprietary dataset with careful curation, and the snippet localization task which is a well-motivated and well-executed idea. The mining pipeline is rigorously validated and the result (MAE 36.3mm vs. 67.0mm) is convincing.
However, several concerns prevent a stronger recommendation.
The retrieval gains are primarily dataset-driven the RefCT-only ablation already surpasses all baselines before any methodological novelty is added and the comparison table is incomplete: Pillar-0 is cited four times and its RATE protocol is directly reused, yet it is absent from Table 1 despite being open source and trivially evaluable.
The ablation table, read carefully, does not fully support the multi-objective narrative: the localization loss shows no effect on all global metrics (reframed as “no degradation”), and the prompt loss slightly but consistently degrades Merlin retrieval, without discussion. Furthermore, the prompt-based disease supervision is never compared against the natural baseline of a standard classification head trained on the same labels, making it unclear whether the AUC gains stem from the design choice or simply from the additional supervision signal.
Acceptance is contingent on the authors providing Pillar-0 numbers in the rebuttal and more carefully scoping their methodological claims.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper’s main contribution is the introduction of RadFinder, a unified 3D CT vision–language model that combines large scale global report–volume contrastive pretraining with prompt based disease supervision and a new intra scan snippet localization objective. The medical staff often link their reports to specific axial slices and this information is now incorporated to guide towards the most relevant slices. This is enabled by building a large dataset of 98k report–volume pairs and automatically mining 262k snippet–slice references from these radiology reports, allowing the model to learn both global semantic alignment and fine grained axial grounding. As a result, RadFinder achieves state of the art retrieval performance, competitive disease classification, and substantially improved slice level localization, all within a single shared embedding space.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Approach incorporating radiologists comments for spotting most relevant slices is nice and pragmatic. The chosen architecture is well structured and explained via Fig. 1. As model strategy processing of 3D volumes is utilized at reduced size of 128x128x32 to allow for resource handling. Here it is questionable, if entire 3D volume processiong for adequate 3D information classification is still the best way to go as redundancy is high and slice information low AND fine grained details might be overseen due to resampling. Thus, recent trends clearly favor full resolution orthogonal slices of selected / chosen quality. But at least in this work slice-level embeddings are incorporated, too RadFinder achieves R@10 = 31.5 on CT RATE, clearly outperforming prior methods such as BrgSA (22.2) and SPECTRE (18.2) under identical evaluation. This is really a very meaningful jump, especially as CT RATE is a difficult benchmark where even marginal improvements can be seen as substantial. RadFinder’s AUC of 83.8 on CT‑RATE and 77.0 on Rad‑ChestCT is essentially tied with the best existing models like MPS‑CT (83.8 / 77.3). Due to that, RadFinder does not significantly advance state of the art in disease classification as the improvements largely come from prompt‑based structured label supervision, a technique already used successfully in 2D VLMs. Compared to high‑resolution slice‑based approaches:
Methods like MedVista3D-ViT or BrgSA use full-resolution orthogonal slices and sometimes detailed local features, which naturally benefit classification tasks that depend on small-scale image details. Given that RadFinder processes downsampled 3D volumes (128×128×32 windows), achieving parity with such methods is respectable but not groundbreaking. RadFinder drops the mean absolute error from 67 mm down to 36 mm, nearly a factor of two improvement over the best baseline (SigLIP2 finetuned). So this is by far the most impressive technical achievement of the paper.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
State-of-the-art presented in the introduction is not very easy to read. Besides, for 3D CT processing, all the approaches that incorporate 2D slices or selected axial/coronal/sagittal slices only are not mentioned, thereby forcing a trade-off between capturing fine-grained details and maintaining the global structural anatomical context, cf. Setio et al. 2016 “Pulmonary nodule detection in ct images: False positive reduction using multi-view convolutional networks”. CLIP-style 2D representation as input for 3D models are recent trends, too (Wang et al. 2022 MedCLIP: Contrastive learning from unpaired medical images and text) or (Lee et al. 2024 “Read like a radiologist: Efficient vision-language model for 3d medical imaging interpretation.”) Paper rather hard to read regarding the writing style. Medical reports/diagnosis could be encoded by international standard such as Snomed-CT instead of the utilized simplified technical binary decisions such as “absence of hepatic lesion”. These could then still be formed as encoding for the classification model but would allow for a standardized taxonomy. RadFinder’s potential and superiority is highest in retrieval and localization, but not in disease classification per se, where full-resolution 2D slice models are really competitive.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Incorporation of radiographer findings is quite innovative and boosts the paper. The writing style itself could be improved to increase clarity.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the reviewers and AC for their thoughtful feedback and address their comments below.
“SPECTRE is trained on A+B+C […] A controlled comparison same data, RadFinder method vs. SPECTRE method is missing.” (R2)
To clarify, SPECTRE was trained on B+C+D (CT-RATE, Merlin, INSPECT), not A+B+C. A=RefCT is our internal dataset and was never available to SPECTRE. The controlled same-data comparison is therefore already in Table 1, row “Public datasets, no loc.” The method-driven gain is substantial on every metric except MAP@5. “RefCT only already achieves R@10 26.3, surpassing all published baselines before adding any of the three training novelties” (R2)
To clarify, the “RefCT dataset” row in Table 1 applies all our training novelties, it just removes the public datasets from the training mix.
“core advantage may come largely from the internal RefCT dataset” (R1) “performance gains seem predominantly dataset-driven” (R2)
Both contribute. The “Public datasets, no loc.” row in Table 1 isolates the method effect, the “RefCT-only” row isolates the data effect. The full model combines both.
“competitors […] fvlm, TotalFM, or Pillar0” (R2)
fVLM is in Table 1 (AUC; the authors do not report CT-RATE retrieval). Pillar-0 and TotalFM publish no comparable retrieval or classification numbers on CT-RATE.
“The implementation reports an effective batch size of 4096 […] which […] implies gradient accumulation” (R2)
The gradient accumulation we use is mathematically equivalent to using a true batch size of 4096: We collect all 4096 embeddings first, then accumulate the gradients with all negatives.
“The prompt loss […] slightly but consistently degrades retrieval on the Merlin benchmark […] This trade-off is not discussed.” (R2)
The prompt loss degrades Merlin results slightly outside the confidence interval while it lifts CT-RATE AUC by 0.27 points and Rad-ChestCT AUC by 0.14 points, a favorable trade-off.
“The prompt-based disease supervision is never compared against a standard classification head trained on the same binary labels. […] Without this baseline, it is unclear whether the AUC gains […] stem from the prompt-in-embedding-space design or simply from the additional disease supervision signal.” (R2)
In Table 1, the “Global loss only” row uses the disease prompts during evaluation, but does not use any of the disease supervision. AUC drops sharply on both CT-RATE and Rad-ChestCT, showing the importance of the supervision signal.
“The added complexity of the prompt approach […] is only justified if it outperforms a simpler linear probe or MLP head on the same encoder” (R2)
A distinct advantage of prompt-based classification over a linear classification head is the potential transferability to new diseases and the possibility to improve the prompts without changing the model.
“All localization evidence appears to be on the authors’ own RefCT test set.” (R1)
We agree. No public CT snippet-slice benchmark exists.
“Medical reports/diagnosis could be encoded by international standard such as Snomed-CT” (R3)
Mapping free-text reports properly to an international standard is a complex topic and out of scope for our paper, where we apply an existing method to structure reports.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper introduced RadFinder, a 3D CT vision–language model that augments contrastive pretraining with prompt-based disease supervision and a novel snippet-slice localization objective, mined from 98k report–volume pairs (262k snippet–slice pairs). Reviewers agreed on the strength of the paper, including novel, well-motivated localization task; clever use of radiologist report references; strong retrieval and localization gains; clean shared embedding space; large-scale data curation. though some weaknesses are also shared by the reviewers, including performance appears largely dataset-driven (private data, not released); classification only ties SOTA; localization loss shows null effect on global metrics; clarity issues. Overall, the reviewers (scores 5/4/5) and AC recommend acceptance, citing the innovative localization idea and substantial empirical results despite concerns about dataset bias and incomplete ablations.