List of Papers Browse by Subject Areas Author List
Abstract
Vision–language models (VLMs) pretrained on large-scale image–text pairs demonstrate strong image-level understanding, but are primarily optimized for global alignment and do not explicitly encode fine-grained anatomical structure, limiting their suitability for spatially precise tasks such as segmentation.
We introduce CheXanatomy, a framework that integrates explicit anatomical knowledge into a pretrained VLM through autoregressive token-space supervision. Instead of adding task-specific decoder heads, the model is trained to generate anatomical segmentation masks via next-token prediction. To enable scalable supervision, we synthesize realistic chest radiographs from CT volumes and forward-project CT segmentation labels to obtain anatomically consistent 2D masks.
We evaluate the approach on synthetic and real chest radiographs against a U-Net baseline, including ablations on model scale, input resolution, and vision encoder fine-tuning. Autoregressive anatomical supervision achieves performance comparable to specialized convolutional models in-distribution and demonstrates improved geometric robustness under domain shift to real CXR data. In addition, anatomy-pretrained models exhibit improved sample efficiency when adapting to novel localization tasks under limited supervision. Larger models and higher input image resolution improve performance, while vision encoder fine-tuning has limited effect.
These results show that embedding anatomical structure directly into the generative objective promotes spatially grounded representations and supports anatomy-aware medical vision–language modeling.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2964_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/sergiosgatidis/CheXanatomy
Link to the Dataset(s)
N/A
BibTex
@InProceedings{GatSer_CheXanatomy_MICCAI2026,
author = { Gatidis, Sergios AND Langlotz, Curtis P. AND Bluethgen, Christian},
title = { { CheXanatomy: Anatomy–Aware Vision–Language Modeling for Chest Radiographs } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution of this paper is an anatomy-aware training framework for pretrained VLMs, where CT-derived anatomical supervision is injected through native token-space autoregressive prediction. The paper further shows that this design improves spatial grounding, geometric robustness, and low-data transfer in chest radiographs.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The proposed framework is technically coherent, with a clear integration of CT-derived supervision, tokenized anatomical prediction, and parameter-efficient adaptation of a pretrained VLM. 2.The experimental study is reasonably comprehensive, including in-distribution, out-of-distribution, and low-data transfer settings, as well as ablations on model scale, input resolution, and vision encoder fine-tuning.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The methodological novelty is limited, as the paper mainly integrates existing ideas for a new anatomy-aware VLM setting. 2.The segmentation comparison is limited, with evaluation mainly against a U-Net baseline rather than a broader set of strong CXR segmentation methods. 3.The paper relies heavily on synthetic CT-derived supervision, but does not directly assess the quality of the synthetic data or projected labels. 4.Some important methodological details remain unclear, particularly the VQ-VAE used for mask tokenization, which weakens reproducibility. 5.The presentation could be improved for better clarity and overall coherence. For example, the related work section should summarize the limitations of existing methods more explicitly and connect them more clearly to the contribution of this paper.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper addresses a meaningful problem with a technically coherent framework and generally positive results. However, the overall novelty is limited, as the method mainly combines existing components in a new setting. The practical advantage over specialized segmentation models is also not fully established, and some key methodological details are not described clearly enough.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
I appreciate the clarification that the intended contribution is dense, multi-label, overlapping anatomical mask prediction via next-token generation. However, the manuscript could more clearly explain why this formulation is advantageous over conventional segmentation approaches in this setting, what specific limitations it addresses, and how the results substantiate this claim. The paper would be stronger if it moved beyond describing the token-generation formulation to more clearly justifying why this design is needed for dense, multi-label, overlapping anatomical masks and experimentally supporting that claim.
Review #2
- Please describe the contribution of the paper
The paper introduces CheXanatomy, a method for making a pretrained chest-radiograph vision-language model more anatomy-aware by training it to generate anatomical segmentations and bounding boxes autoregressively in token space, rather than by attaching a conventional pixel-space decoder. The authors use CT-RATE chest CT volumes, generate synthetic PA and lateral radiographs with DRR rendering, forward-project CT-derived labels to obtain 2D masks for 80 anatomical targets, and then fine-tune a PaliGemma-based model with this supervision. The main empirical contribution is the claim that this anatomy-aware token supervision yields segmentation performance close to a supervised U-Net on synthetic data, improves geometric robustness under domain shift to real chest radiographs, and improves low-data transfer to new localization tasks on RadVLM.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper has a clear and coherent central idea and the formulation is easy to follow - Figure 1 explains the full pipeline well.
2.The work makes good use of scalable synthetic supervision. Generating synthetic chest radiographs from CT volumes and projecting segmentation labels is a practical way to avoid the lack of dense CXR annotations, and the scale is meaningful.
3.The out-of-distribution results are encouraging and visually convincing. In Figure 2, the best 10B/448 model is roughly comparable to U-Net on Dice/IoU on real CXRs while showing clearly better Hausdorff distance and Fourier descriptor distance. Figure 3 supports this qualitatively: the VLM outputs look smoother and more anatomically stable, with fewer jagged boundaries and fewer small irregular false-positive regions than the U-Net baseline.
4.The few-shot transfer experiment is strong and relevant. Figure 4 shows that anatomy-pretrained models outperform non-pretrained baselines most clearly in the very low-data regime (0-3% of RadVLM training data), which supports the claim that anatomy supervision provides a useful spatial prior and improves sample efficiency for downstream localization tasks.
5.The paper includes useful ablations on model size, input resolution, and vision-encoder fine-tuning. These experiments help the reader understand where the gains are coming from, and the finding that higher resolution helps more than vision-encoder fine-tuning is informative for future work in this direction.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The real-world validation is narrow compared with the scope of the claim. The model is trained on 80 anatomical targets, but real-data evaluation is limited to lungs and heart on CheXmask and ribs on VinDr-RibCXR. This leaves it unclear whether the claimed anatomy-aware representation really extends to the broader 80-class setting, especially for small, thin, or heavily overlapping structures. The paper would be stronger with per-structure results and broader real-data validation.
2.The paper sometimes overstates what has been demonstrated about “anatomy-aware medical VLMs.” The experiments show segmentation and improved low-data grounding, but they do not show that the anatomy supervision improves actual multimodal radiology tasks such as report generation, grounded reporting, or clinically meaningful phrase grounding. The paper would benefit from a more careful positioning of what is and is not validated here.
3.The reporting could be more rigorous. Figure 2 is promising, but the paper does not provide statistical testing, confidence intervals, or per-class breakdowns, and the bottom-row plot combines two different real datasets with different structures and annotation styles. That makes the claim of improved robustness harder to interpret precisely.
4.There are also a few presentation issues that should be fixed in revision: Figure 1 uses “Paligemma” instead of “PaliGemma”; the figure shows “Gemma 2 (2B / 9B)” while the text mainly discusses 3B/10B models; there is inconsistent notation for image size (“224 x224”, “224x224”, “448 × 448”); and the sentence around “bounding box IoU Fig. 4” is missing punctuation. These are minor, but polishing them would improve clarity.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper has more strengths compared to weaknesses.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have answered all of my concerns.
Review #3
- Please describe the contribution of the paper
The authors attempt to make a VLM segment X-Rays by supervision trough synthethic 2D anatomy labeling derived from CTs and 3D automatically generated masks. The existing PaliGemma VLM continues to learn to auto regressively generate special tokens with embeddings that map back to the image to produce segmentations trough auto encoder decoding, now using the synthethic data generated by the authors. The authors showcase that it’s possible to adapt PaliGemma for X-ray segmentation by only using performance-efficient fine-tuning (LoRA), without additional major architectural shifts.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper has a very interesting objective: “distilling” the total segmentator into 2D synthetic X-rays, without using more traditional segmentation methods on the MIC literature. The authors demonstrate “it works!”, even if results are not very impressive yet, its a first step and can bring interesting discussions on architectural direction on the field.
Of highlight is the analysis of data fraction in results being favorable to the autoregressive VLM training.
Writing is very streamlined and easy to follow, going directly to the point.
The synthethic data generated on this work could be of value to the community (together with the promised code) but there is no mention of public sharing of the generated x-rays and masks.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Although the idea is very interesting and demonstrated to work, results are not very impressive, especially considering the more than 10-year-old basic UNet baseline. Comparisons with other methods (representative of other architectural strategies) are necessary such as nnUNet and MEDSam.
Especially concerning is the very high result variance displayed by the boxplots.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I am borderline on this. I find the architectural directions explored by the authors very interesting for broader discussions, but the results are not very good in terms of performance, with very high metric variance and limited comparisons with other methods.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors clarified my two main concerns, and informed a more clear scope definition on the manuscript, and justified the high variance in results. More stratified result presentation would be welcome.
In my opinion experiments could be done differently to avoid the necessity of a multichannel overlapping segmentation approach which was used as justification for not using traditional nnUNet. In any case, its my opinion that the paper deserves being discussed in MICCAI due to the very interesting adaptation of VLMs for segmentation.
Author Feedback
We thank the reviewers for their constructive feedback. Since several comments recur across reviews, we group responses thematically. We will revise our manuscript accordingly including correction of formatting issues and legends(R2.4).
Contribution/scope. The central contribution of CheXanatomy is formulating projected CXR anatomy as native autoregressive VLM generation, where dense, multi-label, overlapping anatomical masks are predicted through next-token generation without an external segmentation module(R1.1). This differs from referred-object VLM segmentation, which handles a single object per query rather than dense, overlapping multi-label structures. CheXanatomy is not designed to set segmentation SOTA or replace specialized segmentation models; we will clarify this in the revision(R1.2,R2.2,R3.1).
Baselines. We considered alternatives, including adapted nnU-Net with multi-channel outputs, since nnU-Net is not natively designed for large-scale overlapping masks(R1.2,R3.1). In preliminary evaluation, this was suboptimal for our highly overlapping multi-label task(R1.2,R3.1). We therefore used a specialized multi-channel U-Net, which matched the task and provided a strong baseline(R1.2,R3.1). MedSAM is an interactive bounding box-promptable model that differs fundamentally from autoregressive multi-label dense generation from a text query, and is therefore not directly comparable(R3.1). We will expand the discussion of alternatives (e.g., U-Net++) and clarify that the goal is not exhaustive benchmarking but evaluating token-space anatomy supervision against a task-matched multi-channel baseline(R1.2,R3.1).
Synthetic-label validity. We will clarify that experienced radiologists reviewed TotalSegmentator outputs, DRR plausibility, mask alignment, overt segmentation failures, and ambiguities from overlapping projected structures(R1.3). We will add representative overlays and clarify the review process(R1.3).
Real-world validation. We evaluated publicly available CXR datasets we identified that provide anatomical dense masks relevant to our targets (CheXmask and RibCXR)(R2.1). This covers both compact organs and overlapping structures, but remains narrower than the 80-class synthetic setting because such fine-grained real CXR anatomy datasets are very limited(R2.1). We will state this limitation and clarify that CheXmask labels are automatically generated pseudo-labels rather than purely manual ground truth(R2.1,R2.3).
RadVLM and claim scope. The RadVLM experiments provide complementary evidence for spatial transfer beyond the segmentation labels(R2.2). However, bounding-box localization is not dense segmentation(R2.2). We will present RadVLM as evidence for improved low-data spatial transfer, not validation of all 80 segmentation classes(R2.1,R2.2). We will also scope claims about anatomy-aware VLMs more carefully: the experiments show improved segmentation robustness and low-data localization transfer, not improved report generation or grounded reporting(R2.2).
Reproducibility. We will release the CheXanatomy codebase with the camera-ready version, including prompts, configs, scripts, and data splits(R1.4,R3.3). This includes the pipeline to recreate synthetic data from the publicly available CT-RATE data(R3.3). The VQ-VAE tokenizer is publicly available as part of PaliGemma and will be integrated into our released code; we will also report mask reconstruction details and metrics(R1.4).
Reporting and variance. We will separate CheXmask and RibCXR results to avoid combining datasets with different structures and styles(R2.3). We will also add per-structure breakdowns, confidence intervals, and statistical comparisons where appropriate(R2.1,R2.3). This will clarify that the observed aggregate variance primarily reflects systematic differences in segmentation accuracy across anatomical targets (e.g. heart is easier to segment than ribs), rather than high within-structure variability or instability of outputs(R3.2).
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents an interesting and coherent direction for injecting anatomical supervision into pretrained vision-language models using CT-derived synthetic chest radiographs and token-space autoregressive segmentation. However, the current evaluation is not yet strong enough to support acceptance. The segmentation comparison is limited mainly to a basic U-Net baseline, while stronger and more representative CXR segmentation baselines such as nnU-Net, U-Net++, DeepLab, or other CT-projection-supervision methods are missing. In addition, the reliability of the synthetic supervision pipeline is not sufficiently validated: the paper does not directly assess DRR image quality, projected-mask accuracy, TotalSegmentator error propagation, or conflicts from overlapping anatomical structures. Key VQ-VAE details and reconstruction metrics are also missing, which weakens reproducibility. The reported results show high variance, but no statistical significance testing or confidence intervals are provided. Finally, the nature of CheXmask labels should be clarified, since they are not purely manual ground truth, and the authors should specify whether synthetic CXRs, projected masks, VQ-VAE components, prompts, configs, scripts, and data splits will be released.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
The post-rebuttal opinions are mixed but lean positive, with two reviewers recommending acceptance and one reviewer maintaining rejection. The negative reviewer remains concerned that the paper does not sufficiently justify why token-space autoregressive generation is preferable to conventional segmentation methods for dense, multi-label, overlapping anatomical masks. However, the other two reviewers state that their main concerns were addressed, including clearer scope definition and explanation of result variance. Overall, the work presents an interesting and discussion-worthy direction for anatomy-aware adaptation of VLMs to CXR segmentation using CT-derived synthetic supervision. Remaining limitations regarding stronger segmentation baselines, synthetic-label validation, VQ-VAE details, statistical analysis, and claim scope should be clearly addressed in the final version. The authors should provide the full open-source implementation to improve the reproducibility.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes a coherent approach for injecting CT-derived anatomical supervision into chest radiograph vision-language models via autoregressive token-space generation of dense, multi-label, overlapping anatomical masks. Initial concerns focused on limited segmentation baselines, synthetic-label validity, high variance, and incomplete reproducibility details.
Two reviewers who initially recommended weak reject moved to accept, and the authors clarified the scope: the work is not intended to establish segmentation SOTA, but to explore anatomy-aware supervision for VLMs. The rebuttal also clarifies synthetic-label review, the pseudo-label nature of CheXmask, release plans, and a more careful presentation of real-world validation and variance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Reviewers 2 and 3 support acceptance after rebuttal. Although Reviewer 1 remains concerned that the autoregressive formulation is not sufficiently justified over conventional segmentation approaches, I view the contribution as sufficiently novel and discussion-worthy for acceptance.
