List of Papers Browse by Subject Areas Author List
Abstract
Developing robust 3D-CT foundation models (FMs) using large-scale, real-world clinical data is hindered by three barriers: unstructured report curation, the resolution-batch size trade-off, and vision-language correspondence ambiguity. We propose TotalFM, an organ-separated framework that automatically converts raw clinical CT images and reports into FM-optimized, organ-level volume–finding pairs. By decomposing whole CT volumes into anatomically meaningful units, this approach enables high-resolution 3D encoding with large batch sizes for efficient contrastive learning. The proposed framework was trained on 286,667 CT series and 401,501 organ-level volume-finding pairs. In zero-shot tasks, it consistently outperformed state-of-the-art models in organ-wise anomaly detection and finding-wise AUROC across the majority of disease categories. These results demonstrate that organ-separated learning is a scalable, clinically interpretable principle for bridging the gap between raw clinical archives and 3D-CT FMs. The source code
and pretrained models are publicly available at https://github.com/jichilabo/
TotalFM.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0127_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/jichi-labo/TotalFM
Link to the Dataset(s)
N/A
BibTex
@InProceedings{YamKoh_TotalFM_MICCAI2026,
author = { Yamamoto, Kohei AND Kikuchi, Tomohiro},
title = { { TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper mainly proposes a fully automated pipeline that converts multi-institutional, real-world CT images and radiology reports into learning-ready, organ-level data for foundation model training.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper is generally well written and clearly structured, making it easy to follow. 2.It presents an automated pipeline for handling large-scale real-world clinical data, which is a timely and important direction for developing medical foundation models.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Limited technical contribution. The proposed pipeline primarily integrates existing methods, such as TotalSegmentator for organ segmentation and GPT-OSS for report parsing, without introducing novel methodological innovations. As a result, the technical contribution of the paper appears incremental rather than fundamentally new. 2.Potentially unfair comparison. The evaluation protocol is not entirely fair. Specifically, CT-CLIP and Merlin are evaluated using full CT volumes and English prompts translated from the original Japanese reports, whereas TotalFM is evaluated using organ-cropped CT volumes and original Japanese text. This discrepancy introduces multiple confounding factors, including differences in input granularity and potential information loss or ambiguity from translation, which may bias the comparison in favor of the proposed method. In addition, the evaluation itself is conducted at the organ level, which naturally favors the proposed organ-separated framework and further limits the fairness of the comparison. 3.Lack of analysis on resolution vs. batch size trade-off. A key motivation of the paper is that organ-separated learning alleviates the trade-off between spatial resolution and batch size. However, this claim is not empirically validated. There is no ablation study or controlled comparison demonstrating how organ-level decomposition improves efficiency or performance relative to whole-volume learning, which weakens the support for this central argument and make the paper look incomplete. 4.Limited evaluation task. The evaluation focuses mainly on zero-shot classification tasks. Additional downstream tasks (e. g. , retrieval, localization, or fine-tuning performance) would provide a more comprehensive assessment of the learned representations and better demonstrate the general utility of the proposed foundation model.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My overall rating is primarily based on the limited technical novelty, concerns regarding the fairness of the comparisons, and the lack of experimental validation for the claimed advantages of organ-separated learning. The paper appears somewhat incomplete, as key claims are not empirically supported and the experimental evaluation is limited. A more comprehensive evaluation of the proposed foundation model is necessary to fully demonstrate its effectiveness and generalizability across diverse tasks.
While the paper addresses an important problem and presents a practical pipeline for leveraging real-world clinical data, stronger methodological contributions and more rigorous experimental analysis are needed to convincingly justify its impact.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
I appreciate the authors’ rebuttal and the additional implementation details, which clarify the construction pipeline and training setup. However, my main concerns remain insufficiently addressed. The technical contribution still appears incremental, as the framework mainly integrates existing segmentation and LLM-based parsing tools. The comparison also remains potentially unfair because the competing methods and TotalFM are evaluated with different image inputs and text conditions, introducing confounding factors. In addition, the key claim that organ-separated learning alleviates the resolution–batch size trade-off is acknowledged but not empirically validated through controlled ablation. The evaluation is also still limited mainly to zero-shot classification. Therefore, I do not think the rebuttal sufficiently resolves my concerns, and I maintain my original rating.
Review #2
- Please describe the contribution of the paper
This paper proposes a vision-language foundation model pre-training framework for CT.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
By adopting an organ-separated learning strategy, the method refines the commonly used volume-level modeling into a more anatomically meaningful and efficient organ-level modeling scheme. The overall idea is interesting and practically relevant, and the reported results are encouraging.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Insufficient implementation details. A substantial part of the contribution relies on the automated dataset construction pipeline and the organ-separated training mechanism. Therefore, the implementation details are important for understanding and reproducing the work. At present, the descriptions of the pipeline, cropping strategy, negative sample construction, and training configuration are still somewhat limited. I encourage the authors to provide more complete methodological details.
The motivation for using VideoMAE and the InternVideo2-style training pipeline is not sufficiently clear. CT is a 3D volumetric modality rather than a video modality, yet the first-stage self-supervised pretraining is based on VideoMAE. It is unclear whether the z-axis is effectively treated as the temporal dimension, and if so, the rationale should be explained more explicitly. The spatial continuity along the CT z-axis is not equivalent to temporal continuity in videos. Similarly, the motivation for adopting an InternVideo2-inspired vision-language alignment pipeline for organ-level CT data should also be clarified.
The comparison with Merlin would benefit from a more thorough discussion. The proposed framework does not consistently outperform Merlin across all tasks. The paper mainly attributes this to the fact that the test data are internal to Merlin, but this explanation seems insufficient on its own. Considering that TotalFM is trained on a substantially larger dataset, it would be helpful if the authors could provide a more detailed analysis of the performance gap, as well as discuss the potential limitations of organ-separated learning.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, I think this is a solid paper, but it still has some weaknesses. The main issues are the insufficient implementation details and the lack of justification for directly adopting VideoMAE and an InternVideo2-style pipeline in this CT setting. I believe the paper is potentially above the acceptance threshold, but these concerns should be better clarified.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I remain somewhat concerned about the transparency of the automatic data construction process and the incomplete analysis of organ-separated learning versus holistic modeling. However, compared with the initial submission, the rebuttal provides meaningful clarification on implementation and architectural motivation. On balance, I lean slightly positive, provided that the final version includes the missing methodological details and a more explicit discussion of limitations.
Review #3
- Please describe the contribution of the paper
This paper proposes an approach to train a 3D-CT foundation model from very large dataset derived from routine clinical radiology data. The dataset construction pipeline allows to match clinical findings from the report to the corresponding organ and CT series in the CT images included in the examination. By using TotalSegmentator it allows to localize the organs, pair it with the report findings and train the foundational model from organ-level volume-finding pairs. This allows better handling of GPU memory and to increase the batch size. The authors present results on an internal held out dataset as well as on an external dataset that has never been seen by their model. They evaluate their model on two downstream zero-shot tasks related to organ-wise and finding-wise lesion classification. The results show significant improvement compared to another approach (Merlin) on their held out dataset and slightly lower performance on the external Merlin data (internal dataset of the approach they compare to). I also highly appreciate the effort done on highlighting the limitations of the work in Section 4.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is clearly written and easy to follow. It is also well illustrated and helps understand the content. The evaluation protocol is strong, with a very large dataset (>119k studies) from multiple institutions and a held out test set that was never seen during training. It is also interesting to have tested on an external test and compared to another approach. The solution proposed of separating the training by organ is very interesting and offers memory handling advantages. The results on zero-shot finding classification performance shows that the approach is promising.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
While the organ-separated approach is very interesting and allows to user bigger batch sizes, the results do not really convince me that this improves the quality of the embeddings overall and would really improve downstream tasks significantly. While not stating it directly, the authors seem to base their work on the assumption that the quality of the embeddings (embedded clinical relevance) is highly correlated with higher resolution. But passed a certain resolution, is that really the case? Is there any proof of that?
Did the authors evaluate the quality of translating Merlin reports to Japanese? How did they ensure that the drop in accuracy on their internal dataset is not related to translation errors?
“Merlin Abdominal CT Dataset” => missing ref, how does it differ from the Japanese J-MID dataset? (different country? different pathologies? different CT protocols?)
It is not very clear how CT series are classified in contrast/non contrast for the pairing.
I do not understand the negative augmentation, what is it used for? Why strengthening the learning of normal anatomy?
The use of InterVideo2 for model architecture and training is not justified. Why use an approach for videos on 3D imaging while there is no temporal information and why that one?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper offers a novel approach for training a foundation model from very large 3D-CT dataset from image+report. It offers a very practical solution for leveraging clinical data as is without to much data preparation or cleaning. The solution proposed of training organ by organ is also very interesting and makes sense clinically most of the time (sometimes some findings can be on multiple organs or at the interface etc.). To me this is a preliminary work that should be further explored and the authors should give more insights on why this should work and how to improve it further. How can the clinical relevance be improved? Spending more work on understanding the limitations of the embeddings trained would be very relevant here. I invite the authors to reflect more on the clinical limitations of the approach and push this research a step further.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Despite a few limitations in terms of evaluation of the workflow and a few biases I still find this work relevant to share at MICCAI. It offers a pragmatical approach to batch size limitation that is worth sharing.
Author Feedback
We sincerely thank the reviewers for their valuable comments. Our responses and clarifications are provided below: 1.Implementation Details. (R2.1, R3.4, R3.5) ◦ Dataset construction pipeline: The goal of this pipeline is to construct volume–finding pairs (VFPs), where each pair consists of an organ volume and a corresponding finding sentence. The key factors for matching are the target organ and the contrast phase. On the text side, radiology reports are first segmented into finding-level sentences using a BERT-based classifier. For each finding, an LLM is used to select the corresponding organ and the most appropriate contrast phase. On the CT side, organ segmentation is performed using TotalSegmentator, and the contrast phase is inferred using the totalseg_get_phase option. These components are combined to construct VFPs. ◦ Cropping strategy: For the H and W directions (axial plane), volumes smaller than 192 × 192 were padded outward to match the target size, while larger volumes were resized to the target resolution. Along the Z direction (longitudinal axis), 32 consecutive slices were uniformly sampled. When the organ extent exceeded 32 slices, random sampling was applied during training, and a sliding-window strategy was used at inference to ensure full coverage. ◦ Negative sample construction: Negative samples are introduced to mitigate bias toward disease-positive findings and to increase training diversity. In radiology reports, positive findings are more frequently described (~60% of VFPs), and certain organs are more likely to be explicitly mentioned. We therefore construct negative VFPs by pairing organs not explicitly mentioned in the corresponding report with rule-based normal sentences. ◦ Training configuration: We use AdamW with cosine decay. The base learning rate is 1.5e-4 (pretraining) and 1e-4 (contrastive learning), with weight decay of 0.05 / 1e-4, respectively. Training is performed for 20 / 24 epochs with gradient accumulation of 2.2.Model Architecture (R2.2, R3.6) ◦ We agree that CT data do not exhibit temporal dynamics, and we do not assume temporal continuity. We adopt the InternVideo2/VideoMAE-style pipeline only at the level of 3D token modeling and masked token reconstruction, while adapting it to volumetric CT by treating the third axis as an ordered spatial axis. To avoid imposing a video-specific temporal prior, we adopt factorized axis-wise positional embeddings over the three spatial axes, following a design similar to that used in RadFM. 3.Comparison with Merlin (R1.2, R2.3, R3.2) ◦ For the evaluation on the Merlin test dataset, the LLM-based translations were randomly reviewed by a radiologist. Although not perfect, the semantic content was confirmed to be largely preserved. However, we believe that domain shift arises not only from translation quality, but also from differences in reporting practices and clinical conventions. Similarly, for the imaging data, differences in patient populations, scanners, and acquisition protocols likely contributed to the performance gap observed for TotalFM. 4.Resolution vs. batch size (R1.3, R3.1) ◦ We agree that a controlled ablation study would be valuable for a more rigorous analysis of the resolution–batch size trade-off. However, our claim is not that higher resolution universally improves representations beyond a certain point. Rather, our motivation is that many existing 3D foundation model approaches sacrifice spatial resolution in order to encode full CT volumes within GPU memory constraints, which also limits feasible batch sizes. ◦ Our organ-separated framework is intended as an alternative approach to this trade-off by enabling higher-resolution organ-level representation learning while maintaining practical batch sizes. In addition, larger batch sizes are empirically known to improve contrastive learning stability and representation quality, as discussed in Refs. 5 and 8.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Authors need to clarify the reviewers’ concerns including technical novelty, fairness of the comparisons, insufficient description and justification of the method design, etc.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
My major concern about the paper is: 1.The proposed method integrated existing methods, such as TotalSegmentator for organ segmentation and GPT-OSS for report parsing, without introducing novel methodological innovations. 2.Experimental comparison only included Merlin and CT-CLIP, which is limited and potentially unfair (R1). No ablation study was performed. Therefore, I believe the contribution of the paper does not meets the standards for MICCAI.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
After rebuttal, two reviewers recommend acceptance and the main remaining concerns are limitations rather than fatal flaws. The paper addresses an important and timely MICCAI topic: scalable training of 3D CT vision-language foundation models from routine clinical data. Its main contribution is not a new low-level algorithm, but a practically relevant organ-separated data construction and training framework that enables large-scale organ-level image-report pairing, improves memory handling, and shows promising zero-shot performance on internal and external evaluations.
The concerns raised by Reviewer 1 about comparison fairness, limited ablation, and incomplete validation of the resolution/batch-size argument are valid and should be addressed carefully in the camera-ready version. However, the rebuttal clarified several methodological details and architectural motivations, and the remaining issues can be framed as limitations rather than reasons for rejection. Reviewers 2 and 3 both judged the work to be relevant and worth presenting at MICCAI, especially given the scale of the data, the practical clinical motivation, and the usefulness of sharing this approach with the community.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
the rebuttal helps to clarify implementation details, but it does not resolve the main concerns, especially for R1 retaining the decision for reject. the reviewers asked why this video-inspired is justified, however, the rebuttal gave a plausable adaptation story, but not empirical evidence. Similar, the concern for image resolution vs batch-size, the authors agree that an ablation study would be useful, but only the motivation is restated. This does not desponstrate that the proposed organ-separated framework is better suited than alternatives, which is the core motive of the paper. Furthermore, authors state that randomly reviewing translations by radiologists helps, but the process is vague, i.e., no sample size, no error rate, no structured translation-quality assessment, assessment procedure, etc