Abstract
Medical multimodal models are commonly evaluated with multiple-choice questions by comparing an emitted option label with the reference. We investigate an answer-reporting failure in the Qwen2.5-VL-3B-Instruct (Qwen-3B) pipeline submitted to MedReason 2026 and develop a controlled audit of its dependence on arbitrary option identifiers. The submitted prompt ended with a completed JSON example; changing its answer label alone made the model reproduce that label in 98.0–99.7% of 600 image questions. InternVL3-9B showed the same pattern on 200 cases. We then reassigned labels to fixed option texts and mapped each response back to its denoted option. Two unfilled prompt alternatives increased Qwen-3B accuracy from 19.97% to 23.30% and 26.27%, yet agreement across all five assignments remained only 14.33% and 15.00%. Their relative performance recurred on a non-overlapping, same-source holdout of 600 cases and at a Qwen-7B checkpoint. The completed-example intervention and cyclic reassignment audit identify a failure mode in which the answer reaching benchmark scoring depends on an arbitrary reporting convention, even after the explicit cue is removed. This training-free audit complements single-encoding accuracy with a direct measure of reporting stability. Code and audit scripts are available at https://github.com/chengyi-zhang/the-label-is-not-the-answer.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedReason_007.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/MedReason_007_supp.pdf
Link to Open Review
Open Review Page: https://openreview.net/forum?id=xyiA2hz4HU
BibTex
@InProceedings{ZhaChe_The_MICCAISAT2026,
author = { Zhang, Chengyi AND Bai, Yitong AND Wang, Ziyang},
title = { { The Label Is Not the Answer: Output-Schema Anchoring in Medical Multimodal MCQ Evaluation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17262},
month = {pending},
page = {pending}
}