List of Papers Browse by Subject Areas Author List
Abstract
Vision Language Models (VLMs) are increasingly used for tasks like medical report generation and visual question answering. However, fluent diagnostic text does not guarantee safe visual understanding. In clinical practice, interpretation begins with pre-diagnostic sanity checks: verifying that the input is valid to read (correct modality and anatomy, plausible viewpoint and orientation, and no obvious integrity violations). Existing benchmarks largely assume this step is solved, and therefore miss a critical failure mode: a model can produce plausible narratives even when the input is inconsistent or invalid. We introduce MedObvious, a 1,880-task benchmark that isolates input validation as a set-level consistency capability over small multi-panel image sets: the model must identify whether any panel violates expected coherence. MedObvious spans five progressive tiers, from basic orientation/modality mismatches to clinically motivated anatomy/viewpoint verification and triage-style cues, and includes five evaluation formats to test robustness across interfaces. Evaluating 17 different VLMs, we find that sanity checking remains unreliable: several models hallucinate anomalies on normal (negative-control) inputs, performance degrades when scaling to larger image sets, and measured accuracy varies substantially between multiple-choice and open-ended settings. These results show that pre-diagnostic verification remains unsolved for medical VLMs and should be treated as a distinct, safety-critical capability before deployment.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4379_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/Ufaqkhan/MedObvious
Link to the Dataset(s)
ROCO Dataset: https://github.com/razorx89/roco-dataset
KVASIR Dataset: https://dl.acm.org/doi/abs/10.1145/3083187.3083212
BibTex
@InProceedings{KhaUfa_MedObvious_MICCAI2026,
author = { Khan, Ufaq AND Nawaz, Umair AND Sai Teja, Lekkala AND Saeed, Numan AND Bilal, Muhammad AND Xie, Yutong AND Yaqub, Mohammad AND Khan, Muhammad Haris},
title = { { MedObvious: Exposing the Medical Moravec’s Paradox in VLMs via Clinical Triage } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The main contribution of this paper is the framing of pre-diagnostic visual sanity checking as a separate and clinically important capability for medical VLMs. Rather than focusing on diagnostic text generation, the paper argues that models should first be able to verify whether a small set of medical images is internally coherent and valid for interpretation. the authors introduce MedObvious, a benchmark built around outlier detection in multi-panel image grids, with tasks covering modality, anatomy, viewpoint, orientation, and integrity mismatches, as well as negative controls with no outlier. The experimental study across a range of VLMs suggests that this capability remains far from solved, especially due to false alarms, poorer performance on larger image sets, and prompt-format sensitivity.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper identifies pre-diagnostic visual sanity checking as a distinct capability for medical VLMs, namely verifying that an input is coherent and safe to interpret before generating diagnostic content. This is well motivated clinically and addresses a gap that is largely assumed away in standard medical VLM benchmarks.
2.MedObvious is organized into 1,880 tasks across five progressive tiers and five evaluation formats, and it includes explicit negative controls. The negative-control setting is particularly valuable because it measures false alarms on coherent inputs, which is highly relevant for safety-oriented evaluation.
- The paper evaluates general, medical, and proprietary VLMs in a zero-shot setting, and reports results by format, tier, and positive/negative split. This leads to several useful observations, including false alarms on normal cases, performance degradation with larger image sets, and notable sensitivity to response format.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The main contribution lies in the benchmark formulation and empirical evaluation, rather than in a new model, learning strategy, or inference method. This is still a valid contribution, but it makes the paper primarily a benchmark study rather than a technical/methodological advance.
2.Benchmark validation could be stronger. Although MedObvious is constructed in a template-based way from subsets of ROCO and Kvasir, the paper provides limited detail on quality control, ambiguity analysis, and protection against shortcut cues introduced by the construction process. Additional validation would strengthen confidence that the benchmark truly measures set-level sanity checking rather than dataset-specific artifacts.
3.Clinical realism remains limited. While the motivation is clinically relevant, the actual task is still a simplified 2×2 or 3×3 grid abstraction rather than a full clinical reading workflow. The paper also appears not to be fully polished, with some editorial inconsistencies such as placeholder wording in Table 3 despite numerical results already being shown.
4.Minor issue: Table 3 contains leftover caption text indicating that some results will be added later, although they are already shown.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I scored this paper as Weak Accept mainly because I think the problem it isolates is important for medical VLM evaluation and is not well covered by existing benchmarks. The benchmark is reasonably comprehensive in its task and prompt design, and the experiments produce several practically useful observations, especially around false positives on clean inputs, degradation under distractor scaling, and format sensitivity.
My score is not higher because the paper is primarily a benchmark contribution rather than a methodological advance, and the current benchmark validation still feels somewhat incomplete. Overall, however, I think the problem formulation and empirical findings are strong enough that the paper clears the bar for acceptance, albeit narrowly.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper formally defines the Medical Moravec’s Paradox for medical Vision-Language Models (VLMs), pointing out the critical gap between fluent diagnostic text generation and reliable pre-diagnostic visual sanity checking; it constructs MedObvious, a benchmark with 1,880 tasks covering five progressive difficulty tiers, multiple grid layouts, five evaluation formats and systematic negative control samples, which is specially used to independently test the pre-diagnostic visual triage ability of medical VLMs; in addition, the paper conducts zero-shot evaluation on 17 kinds of VLMs including general open-source models, medical open-source models and proprietary models, and finds that current models still have serious defects in pre-diagnostic verification such as frequent false alarms, performance degradation when scaling to larger image sets and strong format sensitivity.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.It formally proposes and defines the Medical Moravec’s Paradox for medical vision-language models, clearly revealing the critical safety gap between fluent diagnostic text generation and reliable pre-diagnostic visual perception, which fills the theoretical blind spot of existing medical VLM research. 2.It constructs MedObvious, a dedicated benchmark with 1,880 tasks, five progressive difficulty tiers, multiple grid layouts, five evaluation formats and systematic negative controls, which independently assesses pre-diagnostic sanity-checking capability and complements existing medical VLM benchmarks that focus only on diagnostic tasks. 3.It conducts a comprehensive zero-shot evaluation on 17 mainstream VLMs (general open-source, medical open-source, and proprietary models), systematically revealing core flaws like frequent false alarms, scaling degradation, and strong format sensitivity, providing solid empirical evidence for safety risks in clinical deployment.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The evaluation covers a limited number of medical specialties and image types, mainly radiology and endoscopy, lacking data from pathology, ophthalmology, and other fields. 2.The images are organized in a grid format, making it impossible to assess the model’s real ability to comprehend multiple images. 3.The dataset is relatively small and lacks statistical significance.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The dataset is relatively small and lacks statistical significance.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The authors introduce MedObvious, a 1,880-task benchmark designed to evaluate whether Vision-Language Models (VLMs) can perform pre-diagnostic visual sanity checking on medical images. The core premise is that before any diagnostic interpretation, clinicians verify basic input properties — modality, anatomy, orientation, viewpoint, and image integrity — and that VLMs should be held to the same prerequisite. The benchmark presents small image grids (2×2 or 3×3) in which the model must identify an outlier panel or correctly determine that no outlier exists. Tasks are organized into five progressive tiers of increasing clinical specificity (T1–T5) and evaluated across five formats (Detection MCQ/Open, Referring MCQ/Open, and Visual Referring). The authors coin the term “Medical Moravec’s Paradox” to describe the observation that VLMs can generate fluent diagnostic narratives while failing at input-level checks that are trivially easy for humans. Seventeen models (7 general open-source, 4 medical, 6 proprietary) are evaluated zero-shot, with the best achieving 63.2% mean accuracy against a human expert baseline of 88.4%. Key findings include high false-alarm rates on negative controls, performance degradation when scaling from 2×2 to 3×3 grids, and substantial sensitivity to evaluation format.
The benchmark addresses an important and underexplored aspect of medical imaging diagnosis. However, some implementation details remain relatively high-level. More specifically, the exact category balancing and the prompts used — both of which are important for reproducibility — are not sufficiently elaborated in the manuscript.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper addresses a genuine and underexplored gap in medical VLM evaluation. Existing benchmarks (VQA-RAD, PathVQA, SLAKE, PMC-VQA) focus on diagnostic correctness and implicitly assume the input has been correctly perceived. By isolating pre-diagnostic verification as a distinct capability, the authors identify a failure mode that has significant safety implications, particularly for emerging agentic and multi-image VLM workflows. A major strength is that the paper identifies a genuinely underexplored failure mode in medical VLMs: most prior benchmarks focus on answering medical questions or generating diagnostic reports, while MedObvious explicitly isolates the earlier question of whether the input is even valid to interpret. This distinction is clinically meaningful and practically important, especially for safety-critical deployment.
The simplicity of the benchmark is itself a strength. The tasks are designed to be floor-level competencies — distinguishing a CT from an MRI, identifying a flipped image — that any radiology trainee would find trivial. The large gap between human performance (88.4%) and the best model (63.2%) on such elementary tasks makes the central finding striking and difficult to dismiss. The simplicity sharpens the argument: if models cannot pass the floor, more nuanced evaluation is secondary.
The inclusion of systematic negative controls (37.5% of tasks) is well-conceived. Measuring false-alarm rates — the tendency to hallucinate anomalies on consistent inputs — is clinically important and directly relevant to deployment safety. This is a design choice that many benchmarks neglect, and the results reveal meaningful model-level variation in calibration on normal inputs.
The five evaluation formats expose a genuinely important finding about format sensitivity. The large swings between MCQ and open-ended variants of the same underlying task (e.g., Qwen2.5-VL-7B: 75.3% vs. 29.7% on Referring) demonstrate that reported accuracy is strongly interface-dependent, which has direct implications for how benchmark results should be interpreted and how models should be assessed for deployment.
The breadth of models evaluated (17 models spanning open-source, medical-specialist, and proprietary categories) provides a reasonably comprehensive landscape view appropriate for a benchmark introduction paper.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper implicitly assumes that human pre-diagnostic failure modes transfer directly to VLM pipelines — that is, errors a clinician would catch at the sanity-check stage would similarly compromise VLM downstream performance. However, no evidence is presented linking MedObvious performance to downstream diagnostic accuracy. A model that fails grid-based outlier detection might still produce correct reports in its actual inference context, or might fail for entirely different reasons that the benchmark does not capture. Without this linking evidence, the benchmark measures a plausible but unvalidated proxy for deployment safety. At minimum, a correlational analysis between MedObvious scores and downstream task performance (e.g., report accuracy, hallucination rates) would strengthen the safety claims substantially.
The taxonomy of pre-diagnostic checks is borrowed entirely from the clinical workflow without consideration of whether VLMs require a different or expanded set of checks. VLMs may be robust to some human-relevant failures (e.g., laterality markers, given access to DICOM metadata) but brittle to issues with no human analogue: distribution shift relative to training data, resolution or windowing mismatches, photographs of screens versus native DICOM, or acquisition artifacts the model has not encountered. The paper does not discuss or justify why the chosen clinical checks are the appropriate pre-conditioning tests for VLM pipelines specifically.
Some categories — especially the more obvious anatomy, modality, or gross abnormality mismatches — may be driven by relatively coarse visual cues. The paper itself notes that models often recover on the Semantics tier, suggesting that large global appearance differences may be easier than true systematic set comparison. This raises the question of whether benchmark performance is partly influenced by saliency rather than deeper consistency reasoning.
The benchmark scale, while appearing adequate at 1,880 tasks, becomes thin when decomposed across five tiers, five formats, positive/negative splits, and 17 models. Per-condition sample sizes are small — for instance, T5 negative controls evaluated per format yield approximately 21 instances per cell. The paper reports no confidence intervals, significance tests, or variance estimates, meaning that fine-grained comparisons (tier-to-tier degradation patterns, model-to-model ordering, format sensitivity magnitudes for individual models) should be interpreted cautiously. The results are indicative of a problem but do not characterize it with the precision the tables may imply.
Some implementation details remain relatively high-level — for example, the exact category balancing, prompt wording specifics, and details of how template-based construction avoids leakage or unintended shortcuts are not extensively elaborated in the manuscript. A fuller appendix may address these points, but based on the current manuscript text, reproducibility details could be strengthened considerably.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
This paper makes a valuable contribution by revealing that current VLMs — including state- of-the-art proprietary models — fail at pre-diagnostic checks that are trivially easy for humans. The “Medical Moravec’s Paradox” framing is apt, and the benchmark is a useful first step in isolating this failure mode.
Two directions would meaningfully strengthen the work:
First, the most impactful extension would be to validate that MedObvious performance is predictive of downstream safety-relevant failures. Even a small-scale correlation study — showing, for example, that models scoring poorly on specific tiers also make more errors on corresponding diagnostic tasks — would considerably strengthen the claim that this benchmark measures something deployment-relevant rather than an isolated curiosity.
Second, the authors should consider whether the taxonomy of sanity checks needs expansion or revision for VLM-specific failure modes. Clinician checks evolved for human perception; VLMs may need a partially different set of pre-conditioning tests (e.g., distribution shift detection, acquisition protocol verification, input provenance). Discussing this distinction, even without implementing it, would add conceptual depth and better scope the contribution.
The benchmark would also benefit from reporting confidence intervals or bootstrap estimates of accuracy, particularly for the fine-grained per-tier and per-format breakdowns where sample sizes are modest.
These are suggestions for strengthening an already worthwhile contribution, not fundamental objections. The paper opens an important line of inquiry that the community should attend to.
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
This paper makes a valuable contribution by revealing that current VLMs — including state- of-the-art proprietary models — fail at pre-diagnostic checks that are trivially easy for humans. The “Medical Moravec’s Paradox” framing is apt, and the benchmark is a useful first step in isolating this failure mode.
A few directions would meaningfully strengthen the submission:
First, the most impactful extension would be to validate that MedObvious performance is predictive of downstream safety-relevant failures.
The authors should consider whether the taxonomy of sanity checks needs expansion or revision for VLM-specific failure modes. Clinician checks evolved for human perception; VLMs may need a partially different set of pre-conditioning tests (e.g., distribution shift detection, acquisition protocol verification, input provenance). Discussing this distinction, even without implementing it, would add conceptual depth and better scope the contribution.
The benchmark would also benefit from reporting confidence intervals or bootstrap estimates of accuracy, particularly for the fine-grained per-tier and per-format breakdowns where sample sizes are modest.
However, the main issue that is influencing the paper’s score is reproducibility. The exact category balancing and prompt wording are not fully specified, which are both critical for independent replication. The details of how template-based construction avoids leakage or unintended shortcuts need to be elaborated. The authors’ do not explicitly commit to releasing the benchmark dataset, code, or generation scripts, which would be essential for reproducibility and community adoption.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The main issue that is influencing the paper’s score is reproducibility. The exact category balancing and prompt wording are not fully specified, which are both critical for independent replication. The details of how template-based construction avoids leakage or unintended shortcuts need to be elaborated. The authors’ do not explicitly commit to releasing the benchmark dataset, code, or generation scripts, which would be essential for reproducibility and community adoption.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have provided what I consider to be an honest rebuttal that addresses some concerns more convincingly than others.
Some specific comments: the addition of Wilson-score confidence intervals and McNemar tests resolves the statistical precision concern satisfactorily. I don’t think the clinical realism defence they offer engages fully with the distinction between template-generated violations and naturally occurring QC failures, but I think it’s adequate for a first benchmark in this domain. I’m not convinced with the discussion regarding the downstream performance linkage and VLM-specific taxonomy responses.
The central contribution, however, remains of value: The human–model gap on tasks that are easy for clinicians is a topic that needs further exploration and the results of this paper should be exposed to the MICCAI community. I think the work is well-positioned to stimulate further research into the need for pre-diagnostic verification as a distinct capability layer. I recommend accept.
Author Feedback
We thank reviewers for their thoughtful & constructive comments. We are encouraged that novelty & importance of problem were recognized. R1.1/4: Benchmark scope. MedObvious (M-Obv) is a first-ever benchmark & evaluation study for pre-diagnostic visual sanity checking(Sec.1). The central goal is to identify & operationalize a missing safety-critical capability: verifying whether an input is coherent & safe to interpret before any diagnostic reasoning begins. Tab 3 typo has been corrected.
R1.2: Benchmark validation. M-Obv is not formed by arbitrarily mixing ROCO & Kvasir. Each task uses reference-category controlled sampling(Sec.2): inliers come from the same modality, anatomy, and viewpoint-defined category, and only one factor is changed to create the outlier, or none for neg. controls. This reduces shortcut solutions from unrelated source differences and makes the intended consistency violation main signal. The large pos/neg gap, together with scaling degradation and interface sensitivity, further suggests that M-Obv probes set-level sanity checking rather than simple dataset artifacts.
R1.3/R2.1/R2.2 Clinical realism, specialty coverage & grid format. R2 views the grid as a simplification of multi-image reasoning, but it closely match how studies are actually displayed. PACS hanging protocols, 3D Slicer, MRI/CT scans, & multi-view ultrasound all present cases in fixed multi-panel layouts, so panel-level coherence is exactly what a deployed medical system must verify. Likewise, longitudinal assessment requires checking consistency and change across timepoints, and our grid is a controlled abstraction of this requirement. Our 5 modalities already cover the main visual contrasts: gray/color, 2D&tomographic, macroscopic/microscopic. Other specialties, such as pathology&ophthalmology are outside current scope and are extension for future. As R3 notes, the human–model gap (88.4% vs 63.2%) shows that even this floor is not yet solved. We will also include per-modality results & a clearer discussion of generalization in final draft.
R2.3/R3.4: Dataset size/stats. precision. M-Obv is a focused stress test of pre-diagnostic sanity checking, not a pre-training-scale corpus. Related benchmarks are also modest in size, e.g: VQA-RAD (315 images) and GTA (229 tasks). Despite this, 1,880 tasks already reveal large & consistent failures on visually obvious checks. We further support this with Wilson-score 95% CIs and paired exact McNemar tests with Holm correction. Overall accuracy: QwenVL2.5: 63.3 [61.2,65.3], Lingshu: 56.7 [54.6,58.8]. Paired gaps remain significant: d=6.56 pp [4.52,8.53], p=4.6e-7.Thus, M-Obv shows partial VLM progress, but all models remain far below human, so obvious safety checks remain unsolved. We will mention stats. analysis for all models
R3.1: Downstream linkage. M-Obv measures a prerequisite capability that current benchmarks largely assume away, namely, whether the input is coherent enough to interpret at all. A model may still succeed on downstream tasks despite failing this check, but that does not diminish the importance of the prerequisite itself (M-Obv focus). We will further clarify this scope and make it explicit that M-Obv is not a direct predictor of downstream performance
R3.2:VLM-specific checks. VLM may require additional checks beyond those used in human clinical workflows. M-Obv is clinically grounded, radiology-focused subset of pre-diagnostic checks (R1.3) rather than an exhaustive taxonomy for all VLM failure modes. Will add in final version.
R3.3:Coarse saliency. M-Obv test floor-level sanity checks that should be easy for a trustworthy system(Sec.1). The fact that current models still fail substantially on such obvious cases is itself a key result(Tab.2). Same time,M-Obv is not limited to coarse mismatches: the scaling tier, neg. controls, and cross-format evaluation were included to probe set-level comparison beyond simple saliency.We will release M-Obv and its assets upon acceptance
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The reviewers see the paper as a sharp idea on a neglected failure: it cleanly exposes that medical VLMs can speak fluently yet fail at basic pre-diagnostic sanity checks, and the benchmark offers a useful, safety-driven evaluation paradigm with clear empirical signals. But a few major concerns were raised: the work is primarily a benchmark with simplified setups, modest scale, and incomplete validation/reproducibility, leaving open whether these “obvious” failures truly predict real clinical risk or just reflect dataset artifacts. The authors are encouraged to address these comments in the rebuttal.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper introduces a timely and useful benchmark for pre-diagnostic visual sanity checking in medical VLMs. Although the work is primarily a benchmark study rather than a methodological contribution, and the clinical realism and benchmark validation could be stronger, the rebuttal addresses important concerns on statistical testing and reproducibility.
Given the clear human–model gap, the inclusion of negative controls, and the value of exposing this failure mode to the MICCAI community, I recommend acceptance, with the final version clearly discussing limitations around template-based construction, downstream clinical linkage, and VLM-specific failure modes.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper is mainly a benchmark contribution, but the problem formulation is sharp and targets a real human–model gap in “obvious” pre-diagnostic sanity checks. The method formulation and benchmark construction may appear simple, but they do provide useful insight for the community. The rebuttal addresses the main concerns by clarifying construction controls, the scope and diversity of the benchmark, and committing to release the benchmark.
For further improvement, as Reviewer #3 suggested, an extension evaluating the correlation between pre-diagnostic failures and downstream diagnostic failures would be valuable. In addition, the included zero-shot VLM baselines are no longer the most recent, relatively small and generally lack explicit thinking capabilities. More recent large VLMs, including thinking-enabled or domain-specific models, may perform better on these tasks. Overall, the benchmark is useful and timely.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper is accepted as a borderline but valuable benchmark contribution. The reviewers agree that it identifies an important and underexplored problem in medical VLM evaluation, particularly pre-diagnostic verification and false positives on clean inputs. Although concerns remain regarding reproducibility, clinical realism, and the lack of a methodological contribution, the rebuttal added useful statistical analyses and the benchmark is likely to stimulate meaningful discussion in the MICCAI community.