Abstract

Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely transformer-based 3D Diffusion Transformer for volumetric medical image synthesis. Our approach extends diffusion transformers to native 3D data via volumetric patch embeddings and global self-attention that operates directly over 3D tokens. To enable structured control, we propose a timestep-gated control adapter that maps segmentation masks to learnable control tokens, which modulate transformer layers during denoising. This token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformer-based diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5961_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/Cardio-AI/voldit

Link to the Dataset(s)

https://luna16.grand-challenge.org/

BibTex

@InProceedings{SeyMar_VolDiT_MICCAI2026,
        author = { Seyfarth, Marvin AND Dar, Salman Ul Hassan AND Frisch, Yannik AND Wild, Philipp AND Frey, Norbert AND André, Florian AND Engelhardt, Sandy},
        title = { { VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16890},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    1) Fully Transformer-Based 3D Diffusion Model Proposes VolDiT, the first purely transformer-based architecture for 3D medical image synthesis, replacing conventional U-Net backbones.

    2) 3D Tokenization for Volumetric Data Introduces 3D patch embedding to convert volumetric latents into tokens, enabling global self-attention across entire volumes.

    3) Timestep-Gated Control Adapter (TGCA) Develops a conditioning mechanism that injects segmentation-based control tokens with adaptive timestep-dependent gating for stable and precise generation.

    4) Improved Controllability and Anatomical Alignment Achieves better spatial control and anatomical consistency compared to existing diffusion-based methods.

    5) Comprehensive Experimental Validation Shows superior performance on LUNA16 and TaviCT datasets in terms of fidelity, diversity, and conditional accuracy.

    6) Establishes Transformers for 3D Medical Generation Shows that global attention-based models scale effectively to volumetric medical data, providing a foundation for future research.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1) Strong Novelty Introduces the first fully transformer-based 3D diffusion model for medical image synthesis, removing convolutional locality bias.

    2) Effective Global Modeling Uses global self-attention over 3D tokens, improving anatomical consistency and long-range dependencies.

    3) Novel Conditioning Mechanism (TGCA) Proposes timestep-gated control tokens for adaptive and stable conditioning, enabling precise spatial control.

    4) Improved Controllability Achieves better segmentation-aligned generation with improved Dice and boundary accuracy.

    5) Comprehensive Evaluation Extensive experiments on multiple datasets with fidelity, diversity, and conditional metrics, including strong baselines.

    6) Fair and Rigorous Comparisons Uses consistent training settings and includes ablations (model size, patch size, gating).

    7) Scalable and Forward-Looking Design Transformer-based framework is extensible to multimodal and higher-dimensional settings.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1) Limited Novelty Primarily extends existing Diffusion Transformers to 3D; conceptual innovation is incremental.

    2) Dependence on Latent Diffusion Relies on VQ-GAN latent space, similar to prior LDM-based approaches.

    3) Small-Scale Evaluation Experiments are limited to relatively small datasets (LUNA16, TaviCT), raising generalization concerns.

    4) No Clinical Validation Lacks expert evaluation or demonstration in real clinical workflows.

    5) Evaluation Limitations Uses FID and generic feature extractors, which may not fully reflect medical image quality.

    6) Limited Baseline Comparisons Does not compare against more recent large-scale or foundation-model-based methods.

    7) Missing Efficiency Analysis No detailed discussion of computational cost or scalability of 3D transformers.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    1) Strong work overall: Novel transformer-based design with effective conditioning and solid results.

    2) Clarify novelty: Better position against recent transformer/diffusion methods in medical imaging.

    3) Stronger comparisons: Include more recent state-of-the-art or foundation-model baselines.

    4) Evaluation improvement: Consider more medical-specific metrics beyond FID.

    5) Efficiency discussion: Add analysis of computational cost and scalability.

    6) Clinical relevance: Briefly discuss or demonstrate downstream utility.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a solid methodological contribution by introducing a fully transformer-based diffusion model for 3D medical image synthesis. The proposed architecture is well-motivated, and the use of global self-attention for volumetric data is both timely and relevant. In particular, the timestep-gated control adapter is a meaningful addition that enables improved controllability in conditional generation settings. The experimental results are generally strong, showing consistent improvements in fidelity, diversity, and segmentation-aligned synthesis across multiple datasets. The score is moderated by several limitations. The novelty is somewhat incremental, as the work primarily extends existing Diffusion Transformer frameworks to 3D medical imaging. Additionally, the evaluation, while comprehensive in terms of metrics, relies heavily on generic measures and lacks more domain-specific or clinically meaningful validation. The comparisons are also somewhat limited, as more recent large-scale or foundation-model-based approaches are not included. Furthermore, the paper does not sufficiently discuss computational cost and scalability, which are important considerations for transformer-based models in 3D settings.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper proposes CardioDiT, a unified 4D latent diffusion transformer framework for cine CMR synthesis. Unlike previous works that factorize space and time (2D+t), CardioDiT treats the 3D+t volume as a single 4D tensor. It utilizes 4D positional encodings and a spatiotemporal VQ-VAE to model global spatiotemporal dependencies, ensuring inter-slice consistency and physiologically plausible cardiac motion.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Unified 4D Modeling: The move away from architectural factorization (e.g., temporal modules) towards a “native” 4D Transformer is a significant methodological step forward. Consistency Metrics: The introduction of d-SSIM and AR_{ED} provides a rigorous way to measure through-plane anatomical alignment, which is often neglected in synthesis papers. Physiological Fidelity: Validating the model via Ejection Fraction (EF) distributions and W2 distance shows a deep understanding of the clinical requirements of cardiac imaging. Scalability Analysis: The study of model size (S, B, L) and the use of FlashAttention demonstrates the framework’s readiness for large-scale training.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Standardization via Cyclic Repetition: Standardizing sequences to 32 frames via repetition may introduce artificial periodicity. The authors mention this as a limitation, but it does weaken the claim of “truly continuous” dynamics slightly. Dataset Size for Evaluation: While the FID and Precision/Recall are reported, they are calculated on only N=50 synthetic samples. This is a very small sample size for stable FID estimation (standard is usually 2k-50k), which might lead to high variance in reported scores. Temporal Resolution: The model uses a patch size of p_t =2 in the temporal dimension. Discussing how this affects the capture of rapid valve movements or early diastolic filling would be valuable.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    CardioDiT represents the state-of-the-art in 4D medical image synthesis, demonstrating a clear path toward “foundation models” for dynamic medical imaging.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposes a Diffusion Transformer (DiT) framework for 3D medical image synthesis with volumetric semantic mask conditioning. A VQ-GAN compresses input volumes into latents with an 8× downsampling factor per spatial dimension. A two-stage training strategy is adopted: (i) unconditional DiT training using cosine noise scheduling and smooth L1 loss, and (ii) conditional training where the frozen DiT backbone is guided by segmentation-derived control tokens encoded via a 3D CNN. The method aims to address spatial inconsistency issues observed in 3D latent diffusion models (LDMs).

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Introduces a transformer-based denoising architecture for volumetric diffusion, moving beyond 3D U-Net–based LDMs.
    • Clearly motivates the spatial inconsistency limitations of 3D LDMs and demonstrates that mask-conditioned DiTs can partially mitigate these issues.
    • The two-stage training strategy is well-aligned with large-scale diffusion model design principles and allows effective conditioning without retraining the full backbone.
    • Patch-based tokenization with 3D sine–cosine positional embeddings is a reasonable and scalable design choice for volumetric data.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Inconsistent dataset splits are used across evaluations (90/5/5 for Lung CT vs. 60/20/20 for the in-house dataset) without justification, which affects comparability and reproducibility.
    • The evaluation protocol lacks consistency: o TaviCT uses an ImageNet-pretrained 2.5D feature extractor, o LUNA16 uses MedicalNet-pretrained 3D ResNet-50.This results in FID scores differing by orders of magnitude (≈0.005 vs. ≈3.1), making cross-dataset interpretation difficult.
    • Comparisons are limited to 3D LDM baselines. Other relevant 3D diffusion approaches (e.g., wavelet-based or hierarchical CNN diffusion models) are missing.
    • Implementation and system details are insufficient, particularly for a transformer-based volumetric model: o No discussion of hardware, memory consumption, batch sizes, or inference-time constraints. o This is critical given the reported resolutions (e.g., 512×512×256 → 128×128×64 latent volumes).
    • Qualitative analysis is limited: o Figure 3 uses lower-resolution TaviCT volumes (192³) without including corresponding ground-truth images. o It is unclear whether similar inconsistencies persist at higher resolutions such as Lung CT (512×512×256).
    • Tables and figures do not clearly distinguish conditional vs. unconditional generation results, reducing clarity.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a promising and relevant direction for 3D medical image synthesis using Diffusion Transformers with semantic mask control. However, evaluation inconsistencies, missing baselines, and limited reproducibility details substantially weaken the empirical claims. Addressing these issues—particularly by unifying evaluation metrics, adding stronger 3D baselines, and clarifying computational constraints—would significantly strengthen the submission.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

N/A




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper was reviewed by three experts in the field. Based on the reviewers’ feedback, the paper is recommended for provisional acceptance. While the reviewers agree on the contribution of the proposed framework and the promising experimental results, they also raised several valuable concerns that should be addressed in the final camera-ready version, including insufficient implementation details and a lack of efficiency analysis. The authors are encouraged to address these issues to the best of their ability.



back to top