List of Papers Browse by Subject Areas Author List
Abstract
Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce MIS-Ground, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of Medical Image Spatial Grounding. We release MIS-Ground to the public at github.com/asy51/mis-ground. In addition, we present MIS-SemSam, a low-cost, inference-time, and model-agnostic optimization of VLMs that improves their spatial grounding ability with the use of Semantic Sampling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06%.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2003_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/asy51/mis-ground
Link to the Dataset(s)
https://nda.nih.gov/oai
https://github.com/wasserth/totalsegmentator
BibTex
@InProceedings{YuAnd_Medical_MICCAI2026,
author = { Yu, Andrew Seohwan AND Hariri, Mohsen AND Nakamura, Kunio AND Yang, Mingrui AND Li, Xiaojuan AND Chaudhary, Vipin},
title = { { Medical Image Spatial Grounding with Semantic Sampling } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper studies spatial grounding in medical VLMs, focusing on the challenge of localizing anatomical structures in 3D medical images under varying modalities, slice directions, coordinate systems, and medical terminology. It introduces MIS-Ground, a benchmark that systematically evaluates these capabilities across multiple grounding scenarios, including anatomical vs. colloquial direction terms, visual prompts, anatomy priors, and abstract spatial reasoning. The paper also proposes MIS-SemSam, a training-free, model-agnostic inference-time decoding method that improves grounding by aggregating probability mass over semantically related tokens, thereby reducing linguistic brittleness during generation. Experiments on MRI and CT data show that MIS-SemSam consistently improves performance.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A new medical VLM benchmark is contributed to evaluate the grounding capabilities across multiple scenarios.
- The authors introduce semantic sample into medical VLM, which shows promising performance improvement over Qwen3-VL baseline.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The originality and the significance of the introduced MIS-Ground benchmark is unclear. It is not clear whether MIS-Ground is the first benchmarks for understanding of 3D VLM spatial grounding capabilities. If so, this should be clearly highlighted as a key contribution. If not, the authors should clarify its differences from existing benchmarks and methods.
- In page 2, the authors claim that ‘MIS-Ground is designed to identify the extent to which video VLMs are able to identify, localize, and differentiate anatomical structures in 3D medical images’. I am confused as to why this benchmark is tailored specifically for video VLMs, while excluding 3D VLMs and multi-image VLMs.
- Abbreviations ‘RQ’ and ‘AB’ are introduced without prior definition and should be clarified when first mentioned.
- The connection between the utilized method (i. e. , semantic sampling) and the investigated task and benchmark appears tenuous. I am wondering whether semantic sampling also works for 2D grounding or general-domain grounding tasks.
- I am wondering whether semantic sampling is genuinely introduced in this paper, or whether the contribution mainly lies in applying an existing general technique to the medical domain.
- Although semantic sampling appears to be model-agnostic, the current study applies it only to Qwen-3VL. Evaluating it on a broader range of VLMs would make the claims of generality more convincing.
- Qwen3-VL-30B results should not be included in Fig. 2, as their inclusion may be misleading in the current presentation. There is no Qwen3-VL-30B model as claimed in Section 2.3.
- Although several experimental results are provided, there is no insightful analysis and discussions to interpret these results.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this paper introduces a valuable benchmark, it lacks sufficient insightful analysis, and many design choices and implementation details are not well justified.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The rebuttal partially addresses my concerns. However, I feel the novelty of the contributed benchmark is marginal, and analysis provides limited insights to the community. In addition, the writing and the figures (e.g., Qwen3-VL-30B reported in Fig. 2) of this paper should be improved. Thus, I keep my negative rating.
Review #2
- Please describe the contribution of the paper
The paper studies spatial grounding of anatomical structures by VLMs. They propose MIS-Ground, a medical image spatial grounding benchmark containing CT and MRI samples. They also identify language side brittleness in the medical domain with drift into semantically adjacent but incorrect anatomical phrasing as a challenge for current VLMs, and propose the inference-time MIS-SemSam as a mitigation for it.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.It is targeting an important and underexplored area in the medical field, going beyond the general medical VQA and report generation, and studies the abilities of VLMs from a spatial grounding angle 2.The evaluation is very broad; many different models, including Qwen, Molmo, MedGemma, Gemini, and 3D medical VLMs, are benchmarked on their dataset. 3.MIS-SemSam is a nice contribution as it’s inference-time, model-agnostic, and low-cost. Furthermore, the analysis and the theoretical motivation is also good and finally the reported gains on the benchmark are meaningful.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The MIS-Ground benchmark is dominated by CT, leaving how well the overall score reflects general 3D medical grounding performance across modalities. It would be nice to see a better breakdown of how it works on each modality and what are the failure cases. 2.Although the use of MIS-SemSam in the medical terminology and the brittleness of the language modeling is interesting, the method itself is an adaptation of existing semantic sampling to this task, so the methodological novelty is somewhat limited.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
It’s an important and underexplored area at the forefront of the research in this area and the benchmark and analysis and method contributions are good.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
After reviewing the comments by other reviewers, and also that the authors do not address the concern on novelty I am changing my decision
Review #3
- Please describe the contribution of the paper
This paper makes two contributions. First, it introduces MIS-Ground, a benchmark for evaluating spatial grounding of anatomical structures in 3D medical images using video VLMs. Second, it proposes MIS-SemSam, an inference-time semantic sampling strategy intended to improve spatial grounding performance without retraining the model.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The benchmark is the strongest part of the paper. MIS-Ground is well motivated and the design is systematic, covering CT and MRI, multiple slice directions, coordinate system settings, different visual prompts, different text prompts, and different question/answer types.
I also liked the analysis beyond the main leaderboard result. The comparisons between anatomical versus colloquial directional terms, the RAS storage mode setting, and the text-only / white-background ablations provide useful insight into what current VLMs are actually using and where they fail.
The empirical coverage is also a strength. The paper evaluates multiple open-weight VLMs across different sizes, includes medical VLM baselines, and also reports closed-source Gemini results, which makes the benchmark potentially useful for the community beyond this single paper.
MIS-SemSam is also attractive from a practical point of view because it is lightweight, training-free, and does not require changing model weights.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The benchmark contribution is strong, but the methods claim for MIS-SemSam is not yet validated strongly enough. The paper presents it as model-agnostic, but the evidence is still mostly within the Qwen3-VL family rather than across clearly different model families. 2.The headline 13.06% gain is not supported by a direct paired uncertainty analysis between the base model and MIS-SemSam. Since the benchmark contains many correlated questions from the same scans, a simple point estimate is not fully convincing. 3.The method depends on key hyperparameters such as neighborhood size and similarity threshold, but there is no real sensitivity study. This makes it hard to judge robustness and hard for others to know how to use the method. 4.The evaluation omits cases where the expected answer tags are missing. Since formatting failure may differ across models, omission rates should be reported per model to make the comparison fair. 5.The treatment of important baselines is incomplete. In particular, strong closed-source Gemini results are under-discussed, and the very poor performance of M3D and Med3DVLM is not analyzed deeply enough. 6.The aggregate benchmark is still heavily CT-dominant, so it is not fully clear how much the overall conclusions transfer to MRI. 7.More generally, MIS-Ground is valuable, but it is still a benchmark for structured annotation-derived spatial reasoning. The paper should be careful not to overgeneralize this to broad medical grounding or clinical readiness.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I lean weak reject because I find the benchmark contribution meaningful, but I do not think the methods contribution is validated strongly enough for the claims currently being made. In particular, the model-agnostic framing of MIS-SemSam is not sufficiently supported, the main reported gain lacks a direct paired uncertainty analysis, and the handling of omitted outputs and some baselines leaves important questions open. At the same time, I think the paper has value, and a strong rebuttal could improve my assessment.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
After reading the rebuttal, I am moving to Accept. The authors addressed the main points I was concerned about, especially the scope of MIS-Ground as a practical 3D grounding benchmark rather than a video-only benchmark, and the meaning of the model-agnostic claim for MIS-SemSam as an algorithmic property rather than broad empirical validation across many model families. I find these clarifications reasonable.
Some limitations remain. In particular, the MIS-SemSam gain would be more convincing with a scan-level paired uncertainty analysis, and the CT-heavy composition of the benchmark means that the conclusions should not be overgeneralized. However, I do not think these points are strong enough reasons to reject the paper. They are reasonable limitations for an initial benchmark in this direction, and the authors acknowledge part of this in the rebuttal.
Overall, I see the benchmark itself as the main contribution. It is timely, systematic, and publicly released, which makes it useful for the community. I therefore support acceptance, while encouraging the authors to further investigate the methodological questions around MIS-SemSam in future work.
Author Feedback
We thank the reviewers for their thoughtful comments. We focus on the main concerns.
Benchmark novelty/scope (R1,R3). MIS-Ground is not a generic VQA set or leaderboard; it is a controlled 3D spatial-grounding stress test. Its novelty is the factorial design: CT+MRI, 2D/3D inputs, multiple slice directions, RAS storage vs standard viewing orientations, anatomical vs colloquial terms, visual prompts (points/boxes/masks/labels), open/closed/inverted questions, and text-only/white-background ablations. This design isolates failures such as priors dominating visual evidence, visual prompts helping or hurting depending on terminology, and across-slice vs in-plane reasoning. Prior work cited in the manuscript establishes that VLMs fail at medical grounding, but does not provide this 3D, multi-factor, reproducible benchmark for anatomical spatial relations.
Video/3D VLM concern (R1). MIS-Ground does not exclude 3D or multi-image VLMs. Volumes are naturally serialized as ordered slices; therefore video VLMs are a practical input interface. We also evaluated 3D-enabled medical VLMs (M3D, Med3DVLM) and MedGemma. Thus, the benchmark tests medical-image spatial grounding, not a video-only setting.
MIS-SemSam validation/novelty (R1,R3). We agree the core idea relates to Semantic Sampling; our contribution is its VLM/medical adaptation and evaluation: content-token neighborhoods exclude modality/control-token hubs, rescoring is a drop-in final-token replacement, requires no training or extra forward passes, and is tested on identical MIS-Ground questions. We should not overstate “model-agnostic” as empirical proof across all model families; the intended meaning is algorithmic: any VLM exposing logits and embeddings can use it. Empirically, the strongest evidence is in Qwen3-VL, while the benchmark evaluates many other families. This narrower claim still supports the paper’s main contribution.
Statistical/fairness concerns (R3). The +13.06% comparison is directly paired by design: same model, prompts, images, questions, inference settings, and evaluation; only the decoding rule changes. We also report Bayesian credible intervals for model performance. A scan-level paired uncertainty analysis and hyperparameter sensitivity would be useful robustness diagnostics, but refine rather than negate the observed apples-to-apples improvement. Missing answer tags were omitted because automatic grading requires a well-defined answer; this affects instruction-following, not spatial correctness, and the text notes the omission rule.
Interpretation of results (R1,R3). The paper already provides targeted analyses rather than only a leaderboard: RQ1 compares across-slice and in-plane 3D reasoning; RQ2 separates anatomical and colloquial terminology; RQ3 shows when visual prompts help vs conflict with priors; AB1/AB2 separate anatomy prior from abstract spatial reasoning. These analyses explain the central result: current VLM performance is partly visual, partly language-prior-driven, and sensitive to coordinate/terminology choices.
CT/MRI and baselines (R2,R3). The aggregate score is disclosed as CT-weighted (84%) because CT has many more labeled structures; conclusions should be read as benchmark-level, not a claim of clinical readiness or uniform transfer to all modalities. The poor M3D/Med3DVLM results do not dismiss 3D medical VLMs; they show that current 3D medical VLM interfaces are not yet robust to this structured spatial-grounding protocol, especially under resizing and response-format constraints. Gemini is included as a strong closed-source reference, underscoring MIS-Ground’s usefulness beyond our method.
Finally, regarding R1’s “Qwen3-VL-30B” point, no 30B model was evaluated; the evaluated model is Qwen3-VL-32B. RQ/AB denote Research Questions and Ablations. We believe these clarifications address the concerns without altering the submitted experimental basis.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a useful and timely benchmark for medical image spatial grounding, but the method’s contribution is currently under-validated. In particular, MIS-SemSam is claimed to be model-agnostic, yet is mainly demonstrated on Qwen3-VL; the headline gain lacks paired-uncertainty analysis, and missing-tag omissions are not reported per model. The benchmark is also CT-dominant, and the very poor performance of M3D and Med3DVLM is not analyzed in depth enough to distinguish true capability limitations from input/prompt/interface mismatches. I would ask the authors to discuss and address these points.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
I recommend acceptance. The paper introduces MIS-Ground, a controlled 3D spatial grounding benchmark covering CT and MRI with a factorial design across slice directions, RAS storage vs standard viewing orientations, anatomical vs colloquial terminology, visual prompt types, and question forms, together with MIS-SemSam, an inference-time semantic sampling adaptation for medical VLMs. Post-rebuttal, R3 carefully upgraded from weak reject to accept after the authors clarified that the benchmark is not video-only, that 3D-enabled VLMs (M3D, Med3DVLM) and MedGemma were evaluated, and that the model-agnostic claim for MIS-SemSam should be read as an algorithmic property rather than as broad empirical validation. R2 flipped from accept at 5 to reject with confidence 2 by deferring to other reviewers rather than engaging with the rebuttal, so this signal carries little weight. R1 maintains rejection, but the remaining concerns are largely about novelty framing and a figure labeling issue (Qwen3-VL-30B in Fig 2 was a typo for the evaluated Qwen3-VL-32B), and R1’s claim of limited analysis is inconsistent with the RQ1, RQ2, RQ3, AB1, and AB2 analyses already present in the manuscript that R3 explicitly praised. The benchmark itself is the primary contribution: it targets an underexplored capability between segmentation and report-level reasoning, evaluates a broad set of open and closed-source VLMs, and is publicly released. The method’s contribution is more modest as an adaptation than as the introduction of semantic sampling, and it is validated mostly within the Qwen3-VL family. Still, the paper does not need to rest on this. Limitations such as CT dominance and the absence of scan-level paired uncertainty are acknowledged in the rebuttal and are reasonable for an initial benchmark in this direction.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
Although MIS-Ground targets an important problem, the overall contribution remains below the acceptance threshold. The benchmark novelty and insight are disputed, and MIS-SemSam is largely an adaptation with limited validation beyond Qwen3-VL. The headline gain lacks stronger paired uncertainty and sensitivity analysis, the benchmark is CT-dominant, and missing-output handling may affect fairness. With two post-rebuttal reviewers recommending rejection, I recommend rejection.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
After reading the rebuttal, I recommend Reject. The paper studies a useful question: whether current VLMs can ground anatomical structures in 3D medical images under changes in view, coordinate convention, prompt type, and terminology. MIS-Ground is the stronger part of the submission, and its structured settings could be useful for diagnosing failure modes of medical VLMs. However, the paper does not yet make a strong enough case for acceptance.
My main concern is that the benchmark contribution is not positioned sharply enough. The paper should make clear what MIS-Ground adds over existing spatial or medical grounding evaluations, and why its design choices are necessary. The current analysis reports many results, but the discussion does not extract enough insight from them. Important points, such as the CT-heavy composition of the benchmark, the poor performance of medical 3D VLMs, the strong Gemini results, and the omission of malformed outputs, need more careful treatment.
The method contribution is also limited. MIS-SemSam is a training-free semantic sampling strategy, but the paper does not convincingly show that it is a general solution for medical spatial grounding. Most evidence is tied to the Qwen3-VL setting, and the model-agnostic claim is therefore stronger than the experiments support. The reported gains would also be more convincing with paired uncertainty analysis at the scan level, since many questions are correlated within the same scans. Sensitivity to the sampling hyperparameters is not studied in enough detail.
The rebuttal clarifies some scope issues, especially that the benchmark is not meant to be video-only. Still, the main concerns remain: the benchmark novelty is modest, the method is closer to an adaptation of an existing decoding idea, and the submitted paper needs clearer analysis and cleaner presentation. I would therefore keep the paper below the acceptance threshold.