List of Papers Browse by Subject Areas Author List
Abstract
Positron Emission Tomography-Computed Tomography (PET/CT) provides complementary anatomical and metabolic information essential for cancer diagnosis. While vision-language pre-training has shown promise in medical imaging, existing approaches treat PET/CT as a unified modality, failing to capture the distinct characteristics of CT and PET imaging. We propose PETCT-DiVLA, a novel framework that introduces: (1) a token-level modality router that automatically identifies PET-specific, CT-specific, and shared text tokens; (2) a semantic-consistency guided SigLIP loss that leverages organ abnormality labels to define positive and negative pairs dynamically; and (3) dual-granularity alignment at both global and local levels. Our framework employs dual-branch 3D Vision Transformers and achieves mask-free inference. Experiments demonstrate state-of-the-art performance on image-text retrieval and classification tasks, including CT abnormality detection and PET uptake grading. Code is available at https://github.com/Linn60/PETCT-DiVLA.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5633_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Linn60/PETCT-DiVLA
Link to the Dataset(s)
N/A
BibTex
@InProceedings{LuLin_SemanticConsistent_MICCAI2026,
author = { Lu, Lin AND Li, Hang AND Liu, Zihan AND Tang, Chaoxiang AND Zhang, Hui},
title = { { Semantic-Consistent Dual-Image Vision-Language Alignment via Modality Routing for 3D PET/CT } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes PETCT-DiVLA, a vision-language pre-training framework specifically designed for dual-image PET/CT data. The key innovations are: (1) a token-level modality router that classifies text tokens as PET-specific, CT-specific, or shared; (2) a semantic-consistency guided SigLIP loss that uses organ-level abnormality labels to define positive/negative pairs and avoid false negative collisions; and (3) dual-granularity alignment at both global (report-level) and local (organ-level) scales. The framework uses dual-branch 3D ViT encoders and BioClinical-ModernBERT for text encoding. Evaluation is on a large private PET/CT dataset of 8,155 studies.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The problem formulation is original: vision-language pre-training for dual-image (PET+CT) inputs has been underexplored. The observation that existing methods conflate modality-specific information is well-motivated. -The token-level modality router is an elegant solution that naturally decomposes text into PET-relevant, CT-relevant, and shared information without requiring explicit annotation of which tokens belong to which modality. -The semantic-consistency guided loss is well-designed, addressing the false negative problem that is particularly acute in medical VLP where many patients have similar normal findings. -Large-scale evaluation: 8,155 PET/CT studies with 28 anatomical regions provides substantial evaluation scope. The use of TotalSegmentator for organ segmentation and DeepSeek-V3.2 for report parsing is a practical and scalable pipeline. -The t-SNE visualizations (Fig. 2) compellingly demonstrate that modality routing produces well-separated PET and CT subspaces while maintaining intra-modality visual-text alignment
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
-Evaluation is on a single private dataset with Chinese radiology reports. No public benchmark is used, making reproducibility and comparison difficult. -The absolute retrieval performance is quite low (T2I R@1 = 1.25%, I2T R@1 = 1.37%). While the improvements over baselines are substantial in relative terms, the absolute numbers suggest the task may be too difficult or the model needs more data for instance. -The classification results are mixed: for CT abnormality classification, PETCT-DiVLA only marginally outperforms ViT in accuracy (0.562 vs 0.557) and actually has lower F1 (0.319 vs 0.328). The improvements are not clearly statistically significant. -The paper does not compare against the most directly relevant works: Jiao et al. (Ref [4]) and Maqbool et al. (Ref [5]) are cited but not included as baselines, despite being specialized PET/CT VLP methods
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend acceptance because the paper addresses an important and underexplored problem, vision-language pre-training for dual-modality PET/CT data, with a set of well-motivated and technically sound innovations. In particular, the token-level modality router and the semantic-consistency guided loss introduce meaningful advances that directly tackle modality disentanglement and false negative issues in medical VLP. The approach is further supported by large-scale evaluation and convincing qualitative results, indicating that the model learns structured and clinically relevant representations.
That said, there are some limitations, including reliance on a single private dataset, relatively low absolute retrieval performance, and missing comparisons with the most relevant prior work. However, these concerns do not outweigh the paper originality and potential impact.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal is well-written, professional, and addresses the most critical concerns effectively. The R@K and F1 explanations are convincing, and the Refs [4,5] distinction is valid. The private dataset limitation is real but understandable given the field’s data landscape. The core contributions (modality routing, semantic-consistency loss, dual-granularity alignment) remain novel and well-executed.
Review #2
- Please describe the contribution of the paper
The main contribution is describing a new methodology for handling semantic information during vision-to-text interpretation prediction for PET/CT imaging that 1) routes PET and CT-related imaging separately, 2) applies semantic consistency guided loss to mitigate false negative collisions and 3) dual-granularity alignment with mask-free inference abilities. This new methodology is trained and tested in a large cohort of PET/CT imaging datasets, for which we have no clinical insight (eg lymphoma vs metastatic adenocarcinoma etc), and demonstrates improved performance according to a simplified evaluation metric.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Major strengths is development of a handling method that accounts for truly different information that is contained in the CT and PET portions of a PET/CT scan, and which has frequently been not treated as such in prior work. The authors appropriately focus on driving learning from such imaging from distinct CT and PET content, as demonstrated by visualizations of learned embeddings in figure 2.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Major weaknesses include a lack of clinical detail in the datasets used for training and evaluation. For instance, we cannot assess whether the dataset is a truly representative cross-section of patients. It is also challenging to understand for a certain performance metric Recall @ X what this represents. Shouldn’t these values be percentages? Another weakness is that while subjective description of PET intensity can be helpful when trying to predict a radiologists description, and has been performed quite well here, a clinically meaningful model will absolutely need to include the quantitative SUV max value of a suspicious finding in the report. For future studies the authors should consider methods of also retaining the quantitative nature of PET that makes it such a powerful imaging modality.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The authors focus on a legitimate challenge with vision to text prediction for PET/CT imaging, which is a powerful medical imaging modality and where machine learning tools that can speed interpretation can make a large impact in patient care. They demonstrate improved performance with their methodologic approach, and also demonstrate the change to learned embeddings through visualization which suggests their novel methodology is behaving as desired. However, there is also a complete lack of clinical detail on their dataset, difficult clinical interpretability of evaluation metrics, and no discussion of the work’s limitations or weaknesses.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
Rebuttal addresses several comments in the original review–specifically, that source data representing consecutive PET/CT imaging from an oncologic practice and will be delineated in a camera-ready version; that SUV data is not addressed in this work and will be acknowledged as critical for clinical use; and clarification has been provided on the retrieval metrics.
Review #3
- Please describe the contribution of the paper
The paper proposes PETCT-DiVLA, a vision-language alignment framework for 3D PET/CT imaging. Its main contribution is to explicitly model PET and CT as complementary but distinct modalities, rather than treating PET/CT as a single fused input. The method introduces a dual-branch PET/CT visual encoder, a token-level modality router that assigns text tokens to PET-specific, CT-specific, or shared branches, and semantic-consistency guided global/local contrastive alignment for retrieval and classification tasks.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The overall writing and organization are clear, making the motivation, method, and experiments easy to follow. The idea of explicitly modeling PET and CT as complementary modalities is interesting and well motivated, and the proposed framework provides a reasonable direction for improving multimodal representation learning.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper has several weaknesses. (1) The proposed modality router remains a black-box component. Although it assigns tokens into PET, CT, and shared branches, the paper does not verify whether these assignments are semantically correct. Therefore, the router may simply force feature separation or improve optimization, rather than truly aligning PET-related text with PET metabolic features and CT-related text with CT anatomical features. (2) The evaluation does not directly validate modality-specific alignment. The reported retrieval results are computed only between the fused PET/CT embedding and the global text embedding. Higher fused retrieval performance may come from stronger dual-branch fusion, additional supervision, or the router-induced feature separation, rather than successful PET-specific and CT-specific semantic alignment. (3) The evidence from Fig. 2 and the ablation study is insufficient. The t-SNE separation does not necessarily prove semantic alignment, because PET and CT features may naturally separate due to intrinsic imaging differences, and the router itself may also encourage branch-wise separation. Thus, separated clusters do not show that PET-related text is correctly aligned with PET metabolic information or that CT-related text is correctly aligned with CT anatomical information. The ablation study also reports overall gains but does not isolate whether they come from true modality-specific alignment or other factors such as organ-level supervision, semantic-consistency labels, or increased model capacity.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend weak acceptance. The paper is clearly written and proposes an interesting direction for PET/CT vision-language alignment by explicitly modeling PET and CT as complementary modalities. However, the key claim of modality-specific semantic alignment is not fully validated. The router remains a black-box component, and the evaluation mainly uses fused PET/CT embeddings rather than separate PET- or CT-specific alignment. Overall, the work is promising, but stronger targeted validation would better support its main claims.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
After reading the authors’ rebuttal, I maintain my recommendation to accept the paper.
Author Feedback
We thank all three reviewers (R1/R3/R4) for their thoughtful and constructive comments. We address the main concerns below.
1.Modality-specific alignment and router validation (R4-W1/W2/W3). We appreciate R4’s suggestion for clearer justification of the router and stronger validation of modality-specific semantic alignment. Methodologically, the local alignment loss explicitly pairs routed text representations with modality-specific visual branches: t_PET is aligned with visual features extracted from the PET-only encoder, while t_CT is aligned with features from the CT-only encoder. Since these two 3D ViT branches receive different image inputs, this design provides a direct training signal for PET-text and CT-text correspondence, rather than separating features after fusion. As qualitative evidence, in the camera-ready version we will illustrate router behavior using example sentences from our test-set reports (e.g., metabolism-related phrases such as “FDG uptake elevated” receive high PET routing weight, while structure-related phrases such as “soft tissue density nodule” receive high CT routing weight), as a presentation refinement of existing results. The t-SNE in Fig. 2 further shows that t_PET clusters closer to v_PET and t_CT closer to v_CT, consistent with this design.
2.Clinical detail in the dataset and reproducibility (R1-W1, R3). Our data were collected from all consecutive whole-body FDG PET/CT examinations at a single hospital over a 15-month period and include multiple oncologic indications, reflecting a heterogeneous real-world PET/CT population from our institution. We will add cohort-level details in the final version. To our knowledge, no public dataset currently offers paired 3D PET/CT volumes with reports (existing public alternatives are 2D images with reports), which is why public benchmarking is not feasible. To further support reproducibility, we are exploring the possibility of releasing a curated de-identified subset for public research.
3.Recall@K interpretation (R1-W2, R3). R@K values are percentages (e.g., R@1=1.25 means 1.25%). Absolute R@K values are in part determined by the test set size: with 800 held-out cases, R@1 means retrieving the single correct match among 800, intrinsically difficult. This is further compounded by the fact that many sentences in the reports describe normal/healthy findings using highly templated phrasing, making portions of sentence-level content near-identical across reports. In this setting, relative improvement is a more informative indicator than absolute values, and our method shows consistent gains over CLIP and SigLIP across all R@K.
4.F1 in CT classification (R1-W3). On CT abnormality detection our method obtains the best AUC and accuracy. The slightly lower F1 stems from class imbalance: across 28 organs the majority are normal, so F1 is highly threshold sensitive. AUC is the more reliable metric in this setting, where our method achieves a clear improvement. The 5-class PET uptake task is more balanced, and our model improves all three metrics (Accuracy, F1 and AUC). We will clarify this in the final version.
5.Comparison with Refs [4, 5] (R1-W4). Refs [4, 5] and other relevant works are generative models for report generation and are not directly comparable under our evaluation protocol which targets embedding-based retrieval and classification. Their code and data are also not publicly released. CLIP and SigLIP are appropriate vision-language pre-training baselines for our task setting.
6.Quantitative PET properties (R3). We agree that quantitative properties like SUV are clinically important. But preserving exact SUV is typically outside the scope of vision-language pre-training. We will address PET quantification via an agentic post-training method in a follow-up work.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes PETCT-DiVLA, a vision-language pretraining framework tailored for dual-modality 3D PET/CT imaging, addressing the common limitation where prior models treat PET/CT as a single fused modality. The core idea is to explicitly model modality-specific semantics during alignment between images and radiology reports. Although this is a meaningful and non-trivial contribution, but there still exist some gaps in rigor, evaluation, and clarity that prevent a clear accept. I invite authors to rebuttal for addressing the weaknesses depicted in the paper by reviewers.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
remaining points from the reviewers were addressed adequately and all reviewers and myself agree on acceptance of the paper.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Overall, the paper presents an interesting idea of cross-modal vision-language alignment for 3D PET/CT analysis. The authors’ rebuttal clearly addressed the reviewers’ concerns.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes an interesting direction for PET/CT vision-language alignment. All reviewers have recommended acceptance.