Abstract

Recent advances in 3D medical vision–language models have enabled joint reasoning over volumetric images and text, showing strong performance in medical visual question-answering (VQA) and report generation. Despite this progress, it remains unclear whether these models learn spatially grounded anatomy from 3D volumes or rely primarily on learned priors and language correlations. This uncertainty stems from the lack of systematic evaluation of semantic–spatial reasoning in volumetric medical VLMs for clinically reliable decision support. To address this gap, we introduce CT-SpatialVQA, a benchmark designed to evaluate semantic–spatial reasoning in 3D CT data. The benchmark comprises 9077 clinically grounded question-answer (QA) pairs derived directly from 1601 radiology reports and CT volumes, which are validated via a robust LLM-assisted pipeline with a 95\% human consensus agreement rate. Our dataset, requires explicit anatomical localization, laterality awareness, structural comparison, and 3D inter-structure relational reasoning. We also introduce standardized evaluation protocol and benchmark eight 3D medical VLMs, finding severe degradation on semantic-spatial reasoning task, averaging 34\% accuracy and often below random, highlighting the need for deeper integration of volumetric evidence for trustworthy clinical use. The code is available at: \url{https://github.com/BioMedIA-MBZUAI/CT-SpatialVQA}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4615_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/BioMedIA-MBZUAI/CT-SpatialVQA

Link to the Dataset(s)

CT-RATE Dataset: https://github.com/openmedlab/Awesome-Medical-Dataset/blob/main/resources/CT-RATE.md

BibTex

@InProceedings{MonMas_Lost_MICCAI2026,
        author = { Monon, Mashrafi AND Rahman, Umaima AND Hanif, Asif AND Saeed, Numan AND Yaqub, Mohammad},
        title = { { Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision–Language Models } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16895},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a benchmark for evaluating semantic-spatial reasoning in 3D CT data. The main contribution is a dataset comprising 9,077 question–answer pairs derived from radiology reports, designed to assess reasoning capabilities in volumetric medical imaging. In addition, the paper introduces an evaluation protocol and provides a comprehensive assessment of eight 3D medical vision–language models (VLM) on the proposed benchmark.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Valuable benchmark for 3D reasoning: The introduction of a question–answer dataset for 3D CT data is a meaningful contribution, enabling systematic evaluation of semantic and spatial reasoning capabilities in medical vision–language models.

    • Comprehensive empirical evaluation: The benchmark is thoroughly evaluated across multiple (eight) 3D medical VLMs, providing useful insights into the current capabilities and limitations of existing approaches.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Limited differentiation from existing benchmarks: While the proposed dataset is a useful contribution, several related benchmarks (e. g. , [1, 2]) already exist, and it is not clearly demonstrated how the proposed dataset provides additional value or distinct advantages. In particular, prior work such as [1] includes segmentation masks for regions of interest, which offer richer supervision and evaluation capabilities, whereas the proposed dataset does not provide such annotations. This limits its utility for tasks requiring precise spatial grounding.

    • Limited novelty in evaluation protocol: Although the paper presents the evaluation protocol as a key contribution, similar approaches leveraging LLMs to assess generated text have already been explored in prior work (e. g. , [3]).

    [1] Bai, F. , Du, Y. , Huang, T. , Meng, M. Q. -H. , & Zhao, B. (2024). M3D: Advancing 3D medical image analysis with multi-modal large language models.

    [2] Chen, Y. , Xiao, W. , Bassi, P. R. A. S. , Zhou, X. , Er, S. , Hamamci, I. E. , Zhou, Z. , & Yuille, A. (2025). Are vision-language models ready for clinical diagnosis? A 3D medical benchmark for tumor-centric visual question answering.

    [3] Wang et al. , LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation, 2024.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Providing a dataset comprising 3D CT data with associated question–answer pairs is a valuable contribution, and evaluating it across existing VLMs is useful for the community. However, it is not clearly demonstrated how the proposed dataset provides additional value beyond currently available benchmarks (see weaknesses), particularly in terms of added information or evaluation capabilities.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The authors conduct a systematic review of Vision Language Models’ (VLM) ability to answer spatially-grounded questions, based on the imaging data only. They achieve this by creating a novel Visual Question Answers (VQA) dataset and benchmarking 8 state-of-the-art VLM in a zero-shot setting. The results demonstrate that VLM consistently fail at answering spatially-grounded questions.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    This work represents, in my opinion, highly relevant and interesting research for the community, addressing a clearly-defined research question through a well-executed analysis. The work has important implications for the trustworthiness and clinical translation of VLM-based methodologies. A particular strength is the broad panel of benchmarked models, whose consistently similar failure behaviour points to a structural limitation rather than a model-specific shortcoming. The results are clear and supported by large effect sizes.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The paper has several weaknesses, most of which are minor. First, the study does not distinguish whether the failure in spatial grounding stems from the visual component or from the internal semantic representations of the VLMs; a useful control would have been to provide the text of the radiology report as input and verify whether accuracy improves, which would isolate the failure to the visual component. Second, the LLM-as-judge paradigm itself lacks validation in this context; asking a human radiology expert to evaluate a subset of answers would have provided the necessary verification. Third, several methodological details are missing: it is unclear whether the LLM judge was asked for a score or a binary correct/incorrect judgment, what prompt was used for the evaluation, and how the LLM-as-jury results were aggregated; additionally, the text mentions both LLM-as-judge and reference-based evaluations, but only LLM-as-judge results appear to be reported in Table 1.

    Regarding the human audit of the generated QA pairs, the annotators were not provided with the images, which would have allowed them to verify whether the target answers were findable from the visual information alone; furthermore, a useful baseline would have been to measure how an expert human radiologist would score on the same test, as without this it remains possible that some questions were too vague or impossible to answer. Finally, the results lack measures of variability such as confidence intervals or statistical significance tests, though the consistently low and uniform performance across VLMs mitigates the impact of this omission.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    As a minor presentation remark, the caption of Table 1 does not make it clear that the numbers in the top rows represent accuracy, and should be updated accordingly. Additionally, Figure 5 presents a radar plot that is difficult to interpret in its current form; removing the shading would reduce visual clutter, and the authors may wish to consider replacing it with a bar plot or another more readable format.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The overall score reflects the high impact, relevance, and significance of the research question addressed by this work, as well as the clarity of the results and, in particular, the consistent failure mode observed across a broad panel of VLMs.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This work is timely and important for research in the 3D CT vision-language model space. It introduces a new benchmark dataset, CT-SpatialVQA, with spatially grounded question-answer pairs designed to evaluate 3D CT VLMs on spatial grounding and complex inter-structural relationships. The benchmark appears aimed at testing a more nuanced spatial understanding of 3D CT volumes than prior benchmarks.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The benchmark is built from 1,601 report-plus-CT volume pairs and yields 9,077 QA pairs, making it substantially larger than prior benchmarks and is aimed at rigorously testing the spatial grounding of the VLM models.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The benchmark is derived from a single public dataset from one institution and is limited to adult, non-contrast chest CT. This creates a narrow evaluation scope and may limit its value as a more general benchmark for 3D CT VLM assessment. 2.The results do not include comparisons of model performance on other established CT benchmarks which could emphasize the importance of this work and how performance on other benchmarks could be misleading on model’s actual potential. 3.Figure 3 raises concern that some of the example QA pairs are overly leading and may not adequately evaluate true spatial or clinical reasoning. In particular, questions such as whether the ascending aorta is anterior or posterior, or whether consolidation is in the right or left lung, effectively reduce the task to a binary choice, giving the model a substantial chance of answering correctly without demonstrating deeper understanding. This weakens the claim that the benchmark measures nuanced 3D spatial comprehension. A stronger design would use less guided questions, for example asking whether a finding is present and, if so, where it is located within the chest.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    1.Figure 2 (QA Pipeline): Figure 2 is potentially misleading in how it depicts the QA generation pipeline. Please clarify why the 3D CT scans are shown as inputs to the QA generator LLM, since a standard LLM would not directly process volumetric image data unless an intermediate vision or structured representation module is used. 2.Prompt consistency: The prompt for the VQA generator is provider but the prompt used for the VQA validator is not provided, why? Additionally, report metrics on the first LLM QA generator (total pairs generated) and validation LLM filtered counts (number discarded), including common rejection causes. 3.QA Curation Transparency: Only 12% of the QA pairs were manually evaluated. Please provide absolute counts and clarify how they were selected (e.g., stratified across pathologies?). With 95% approval rate, discuss the remaining 5% failure cases, including their nature and implications as limitations. 4.With/Without Image Evaluation: The MIRAGE paper demonstrated that VLMs can generate convincing reports even without image input in many cases. Given the benchmark’s unambiguous but leading questions (often giving models a 50% baseline success rate), consider evaluating models both with and without image input to highlight performance differences. 5.Benchmark Naming: Consider making the chest CT scope explicit in the benchmark name (e.g., “Chest-CT-SpatialVQA”) to better reflect its specific focus and avoid overgeneralization.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Benchmarks that are spatially grounded are a need of the hour for evaluating medical VLM models and this work is a step in the right direction.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We want to thank the reviewers and meta-reviewer for their positive impressions of our work and address the points they raised. We address the main concerns first, then the minor points:

R1 (W1): Differentiation from Existing Benchmarks We respectfully clarify that the referenced benchmarks (M3D and DeepTumorVQA) do not include spatially grounded QA pairs. CT-SpatialVQA specifically evaluates semantic-spatial reasoning through structured, spatially grounded question-answer pairs, a dimension not covered by prior work. We will make this distinction more explicit in the camera-ready version.

R1 (W2): Novelty of Evaluation Protocol We did not claim novelty in the evaluation metrics themselves. The LLM-as-judge approach was adopted as an established and appropriate evaluation strategy for this task. We will state this more clearly in the revised manuscript.

R2 (W1): Visual vs. Semantic Failure Attribution Providing only the question text without an image does not reflect the intended use of CT-SpatialVQA, which is designed to assess visual reasoning grounded in images. However, testing with empty or missing images can serve as a complementary analysis to reveal language priors or biases. We will clarify this distinction in the paper.

R2 (W2): LLM-as-Judge Validation We appreciate this suggestion and will explore radiologist annotations as a validation mechanism in future work.

R2 (W3): Missing Methodological Details The evaluation prompt, judgment format, aggregation strategy, and reference-based evaluation results will all be added in the camera-ready version.

R2 (W4): Human Audit Without Images We will add a clearer description of the human verification process for QA pair construction, including how questions were grounded in the findings and impression sections of radiology reports, to better justify the use of GPT for dataset construction. Radiologist-level scoring will be explored in future work.

R3 (W1): Single-Institution, Single-Modality Scope We acknowledge this limitation. Expanding CT-SpatialVQA to multi-institution and multi-modality settings will be explored in subsequent studies, and we will note this explicitly as a limitation in the camera-ready version.

R3 (W2): Comparison to Other CT Benchmarks We included all 3D CT VLMs for which code and model weights were publicly available. We will note this constraint explicitly as a limitation in the camera-ready version.

R3 (W3): Leading/Binary Questions We clarify that in addition to binary-choice questions, CT-SpatialVQA includes spatial relative positioning questions that are not reducible to a binary answer. We will make this clearer in the paper.

## Optional Comments

R2: Table 1 Caption and Figure 5 We will mention accuracy explicitly in the Table 1 caption. We will also remove the shaded region from the radar plot in Figure 5 for improved clarity.

R3: Figure 2 Misrepresentation Figure 2 will be corrected to clarify that 3D CT volumes are not directly input to the QA Generator LLM; the arrow from CT images to the QA Generator will be removed as it was a typographical error. We will also include prompt details, QA generation statistics, and validation/filtering counts for completeness.

R3: Prompt Consistency and Curation Transparency The prompt used for the VQA validator, absolute counts for the manually evaluated subset, selection criteria (e.g., stratification across pathologies), and a discussion of the 5% failure cases will be added in the camera-ready version.

R3: With/Without Image Evaluation Please refer to R2 (W1).




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper is accepted because it introduces a valuable and timely benchmark for evaluating semantic and spatial reasoning in 3D CT-based medical VLMs.

    The reviewers appreciated the dataset scale, with 1,601 report–CT volume pairs and 9,077 QA pairs, as well as the broad evaluation across eight 3D medical VLMs.

    The results provide clear evidence of consistent failure patterns across models, highlighting important structural limitations and offering meaningful implications for trustworthy clinical translation.



back to top