Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Existing text-to-imaging generation methods inherit diffusion architectures from natural image synthesis, relying on explicit cross-attention to condition generation on text at every step. We hypothesize that this design is unnecessarily complex for medical tomographic volumes: unlike natural scenes, human anatomy exhibits strong structural regularity, with organs occupying predictable locations and varying far less across individuals than open-ended visual content. Given a text-aligned tokenization, sequential modeling alone should therefore suffice for text-conditional anatomical generation, without dedicated cross-attention layers. We propose CTTok, a discrete autoregressive approach that extends the vocabulary of a pre-trained language model with anatomical tokens representing 3D CT patches. Text conditioning is handled by the language model’s existing causal attention over these anatomically grounded tokens. Because the synthesized volume is itself constructed from such tokens, any mild boundary artifacts from patch-level discretization carry no semantic ambiguity and can be removed by a single-pass, unconditional GAN refinement with no access to the original prompt, since all text-to-anatomy correspondence is already resolved at the token level. CTTok outperforms state-of-the-art diffusion and flow-matching baselines in diversity, image quality, and text-image alignment, at a fraction of the training and inference compute. Source code and model weights: \url{https://github.com/WongJiayi/CTTok}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2284_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/WongJiayi/CTTok

Link to the Dataset(s)

CT-RATE Dataset: https://huggingface.co/datasets/ibrahimhamamci/CT-RATE

BibTex

@InProceedings{WanJia_CTTok_MICCAI2026,
        author = { Wang, Jiayi AND Reynaud, Hadrien AND Dombrowski, Mischa AND Wang, Xiaoliang AND Hamamci, Ibrahim E. AND Shit, Suprosanna AND Er, Sezgin AND Menze, Bjoern H. AND Kainz, Bernhard},
        title = { { CTTok: Voxel-Abulary for Autoregressive 3D CT Volume Generation with Large Language Models } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16890},
        month = {September},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    CTTok proposes a discrete autoregressive framework for text-conditioned 3D CT volume generation. The key idea is to reformulate text-to-CT synthesis as language modeling: 3D CT volumes are tokenized into discrete patch-level tokens via a frozen codebook, and a pre-trained LLM (Qwen-1.5B) generates these tokens autoregressively, conditioned on radiology report text through standard causal attention rather than dedicated cross-attention layers. The paper argues that human anatomy follows a predictable “grammar,” making explicit cross-attention unnecessary — text conditioning emerges naturally from the LLM’s existing sequential modeling capabilities. A single-pass 2.5D GAN refines blocking artifacts without requiring any text input

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. The proposal to replace diffusion-based cross-attention conditioning with autoregressive generation over spatially disentangled tokens is a fresh perspective for medical volume generation. Most prior work (MedSyn, GenerateCT, CT-Flow) imports diffusion or flow-matching architectures from natural image synthesis without questioning whether they are necessary for structured anatomical data.

    2.Using random hyperplane hashing over a text-aligned vision encoder to build a frozen atlas of real 3D patches is a smart departure from learned VQ-VAE codebooks. The spatial disentanglement property — each token independently encodes a fixed region with no convolutional coupling through a shared decoder — is a genuine architectural advantage that enables the LLM to treat volume generation as sequence modeling.

    1. Generation in 2.2 seconds per volume versus 84–100 seconds for text-conditioned baselines represents a 40–50× speedup.

    2. The paper reports diversity (IRS), quality (FID), inter-slice continuity (FVD), and text alignment (CLIP) — covering multiple axes that generation papers often evaluate incompletely.

    3. Anonymous code link provided, weights promised by conference time, and implementation details (optimizer choices, training hours, memory footprint) are clearly reported.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1. The “anatomy as grammar” thesis is the paper’s central claim but receives no direct validation. The paper argues that anatomical structure is predictable enough that an LLM’s causal attention suffices for text-to-anatomy grounding. However, the evidence is purely indirect: the method achieves good metrics, therefore the thesis must hold. No probing experiments are provided — for example, whether the LLM’s attention patterns correlate with anatomically relevant text tokens, whether scrambling anatomical terms in prompts causes structured degradation in generation, or how the model handles contradictory or nonsensical prompts. Without such experiments, the thesis remains an appealing narrative rather than a validated scientific claim.

    2. The real contribution may be the corpus and codebook, not the LLM or the grammar thesis. Drawing the paper’s own analogy to language: in NLP, building a strong language model requires a large representative corpus and good tokenization - the model itself learns distributional structure through standard training. CTTok follows the same recipe: a large corpus (CT-RATE, 40K+ volumes), a text-aligned tokenization scheme (SigLIP + random hyperplane hashing), and standard LLM fine-tuning (next-token prediction). Three observations suggest the codebook and corpus are doing the heavy lifting:

    a) The SigLIP encoder is pretrained with text-volume contrastive loss, meaning the feature space is already text-aligned before the codebook is even constructed. Text conditioning may not “emerge from causal attention” so much as it was pre-embedded in the SigLIP features. b) The LLM fine-tuning is entirely standard, with no architectural innovation. The paper does not ablate whether a smaller or different LLM achieves comparable results given the same codebook. c) The GAN refinement operates without text input yet is responsible for converting blocky reconstructions into visually convincing volumes. Without pre-GAN quality metrics, the LLM’s specific contribution to final image quality cannot be isolated. The paper would be substantially strengthened by ablating corpus size, reporting pre-GAN FID, testing a simpler sequence model, and inspecting whether codebook hash buckets correspond to anatomically meaningful structures.

    1. No downstream clinical validation despite strong clinical motivation. The paper motivates itself with three clinical applications: data augmentation, privacy-preserving sharing, and rare-pathology simulation. None are validated experimentally. Does training a downstream model on CTTok-generated data improve performance? Are generated volumes diverse enough to avoid memorization? Can the model generate rare pathologies reliably? FID and CLIP scores do not answer these questions.

    2. MaisiV2 is conditioned on segmentation masks rather than text, making it fundamentally incomparable. GenerateCT is listed as conditioned on “Report” in the table but “metadata” in the footnote — this needs clarification. The most relevant discrete-tokenization baseline for 3D generation (Dai et al. , 2024, cited as reference [3]) is discussed but not benchmarked against. These issues make the quantitative comparison less clean than it appears.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Weak accept. CTTok proposes a genuinely novel approach — autoregressive generation over a frozen atlas codebook, eliminating cross-attention for text-to-CT synthesis — with strong quantitative results and a dramatic speed advantage. The codebook design is creative, the evaluation is thorough, and the paper is clearly written. However, the central “anatomy as grammar” thesis is asserted but not probed, and the paper’s own NLP analogy suggests the real contribution is the corpus and codebook rather than the LLM’s learning: the SigLIP encoder already provides text-anatomy grounding, the LLM fine-tuning is standard, and the GAN’s contribution to final quality is not isolated. Clinical motivation is stated but not validated with downstream experiments.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper proposes CTTok, which reformulates text-to-3D CT generation as an autoregressive language modeling problem. By constructing a frozen anatomical codebook, fine-tuning Qwen-1.5B to generate discrete token sequences, and then refining with a 2.5D GAN, it achieves fast and high-quality CT volume generation.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The method features a novel and elegant design approach, which fully converts 3D CT generation into a language modeling task. It leverages the causal attention inherent in LLM to achieve text conditioning, eliminating the complex cross-attention mechanism in diffusion models. 2.The inference speed is approximately 40 to 50 times higher than that of existing methods, and the source code is provided, demonstrating strong potential for practical applications.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.This paper proposes CTTok, which reformulates text-to-3D CT generation as an autoregressive language modeling problem. By constructing a frozen anatomical codebook, fine-tuning Qwen-1.5B to generate discrete token sequences, and then refining with a 2.5D GAN, it achieves fast and high-quality CT volume generation.

    2.CTTok relies entirely on implicit learning of the LLM without any structural guarantee for anatomical correctness. All evaluation metrics (FID, FVD, IRS, CLIP Score) are distribution-level statistics, and none of them measure the anatomical correctness of generated CT volumes. I suggest adding segmentation consistency evaluation on generated volumes, or validating through downstream task augmentation.

    3.The current generation resolution is 256×256×128, which is below the clinical standard of 512×512.Since CTTok’s sequence length scales proportionally with volume resolution, scaling to higher resolution would significantly increase the number of tokens, limited by the LLM context window. Scalability should be discussed.

    4.In Fig. 3, do the different refinement methods use the same input? If they are not based on the same discrete reconstruction result, the comparison is unfair. I suggest the authors clarify the experimental setup and ensure all methods are compared using identical inputs.

    [1] Yu L, Cheng et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms [2] Chen Z, et al. LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Model

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    CTTok proposes a novel and efficient paradigm for text-to-CT generation by formulating the problem as autoregressive language modeling, achieving strong performance in speed and image quality metrics. However, the core claim about LLM’s anatomical mapping ability lacks verification or discussion, and clinical effectiveness evaluation is insufficient. If the rebuttal can provide additional discussion and validation, it would significantly strengthen the contribution of this paper.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposes CTTok, a discrete autoregressive framework for text-to-3D CT volume generation using a pretrained language model. The method constructs a large anatomical codebook of 3D CT patches and formulates volumetric generation as next-token prediction conditioned on radiology report text. The generated token sequence is reconstructed into a volume and refined using a lightweight 2.5D GAN to reduce discretization artifacts. The authors evaluate the approach on the CT-RATE dataset and report competitive performance in standard generative metrics while achieving substantially faster inference compared to diffusion- and flow-based baselines.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Interesting modeling perspective bridging language models and volumetric medical image generation. The paper presents a conceptually appealing formulation of text-to-CT synthesis as autoregressive sequence modeling over discrete anatomical tokens. Treating volumetric data as token sequences is aligned with recent trends in generative modeling and represents a reasonable alternative to diffusion-based approaches. 2.Strong computational efficiency and practical deployment potential. The proposed framework achieves faster inference compared to diffusion and flow-matching baselines, generating volumes in seconds rather than minutes. This efficiency improvement is practically meaningful for large-scale data augmentation, simulation, and clinical workflow scenarios.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.While the application domain is novel, the overall pipeline follows a familiar pattern of discrete tokenization, autoregressive sequence modeling, and post-hoc refinement. The technical contribution appears to be an adaptation of existing token-based generative modeling paradigms (e.g., VQGAN, MAGVIT, TokenFlow and MaskGIT) to CT volumes rather than a fundamentally new modeling framework. 2.The paper suggests that language models inherently possess the grounding needed to model anatomical structure and that explicit cross-attention mechanisms are unnecessary. These claims are strong but not rigorously validated through ablation studies (e.g., comparisons with standard transformers, different conditioning strategies, or models without pretrained language knowledge). 3.The final image quality depends on a 2.5D GAN refinement step, yet the contribution of this component relative to the autoregressive generator is not clearly quantified. An ablation evaluating performance without refinement would help clarify where the performance gains originate. 4.Evaluation focuses primarily on perceptual metrics rather than clinical or anatomical validity. Metrics such as FID, CLIP score, and diversity capture visual similarity but do not directly measure anatomical correctness or clinical usefulness. Additional evaluation using segmentation consistency, anatomical plausibility, or expert assessment would strengthen the claims about medical relevance.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please refer to the Weaknesses.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers and the Area Chair for their constructive feedback. As an early-accepted paper, we focus our response on clarifying possible misunderstandings and summarizing the corresponding camera-ready revisions. The main concern was the missing pre-GAN evaluation. We have added the missing pre-refinement evaluation details in Table 4 by directly decoding the LLM-generated token sequences before applying the 2.5D GAN refiner. This clarifies the respective roles of the two stages: text alignment is already largely preserved before refinement, while the GAN mainly improves visual quality and reduces inter-patch artifacts. A second concern was the interpretation of the LLM’s anatomical mapping ability. We revised the abstract, introduction, contribution statement, and conclusion to clarify that text-anatomy alignment is supported jointly by the SigLIP-based codebook and the LLM, rather than by the LLM alone. We now describe the LLM’s role as capturing text-to-anatomy correspondences through causal attention over text-aligned anatomical tokens. We also added a concise limitations paragraph acknowledging that our evidence for the LLM’s role is empirical rather than mechanistic, and that attention probing, downstream clinical validation, and higher-resolution scaling remain future work. Finally, we clarified baseline conditioning differences, figure captions, and related work, including the SPAE reference.



Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This paper was reviewed by three experts in the field. Based on the reviewers’ feedback, the paper is recommended for provisional acceptance. While the reviewers agree on the novelty of the proposed framework and the promising experimental results, they also raised several valuable concerns that should be addressed in the final camera-ready version, including the lack of pre-GAN quality metrics and the need for further clarification of the core claim regarding the LLM’s anatomical mapping ability. The authors are encouraged to address these issues to the best of their ability.



back to top