Abstract
When a medical agent calls a vision-language model, prompt text, image bytes, and structured response fields may pass through separate software components. We study a mismatch at this boundary. The prompt says an image is attached, while the retained client request contains no image bytes. A diagnosis in this state can pass a schema check and reach later software without visual evidence. In the baseline contract, GPT-5.4, Claude, and Gemini filled the diagnosis field in 617 of 3,000 calls across five hosted models. Changing only demographic wording also changed the returned diagnosis distribution. In one matched GPT-5.4 chest X-ray pair, the leading diagnosis moved from Pneumothorax in 54 of 100 calls for a profile described as a white man to Sarcoidosis in 77 of 100 after white changed to Black. The broader analysis covers 288 direct age, race-word, and sex-word contrasts. None of 19,350 hosted no-attachment controls filled the diagnosis field. Four public software regressions show how images or task text can disappear at client boundaries. We bind the completed request and selected diagnosis field to a caller-owned record of the task and ordered image fingerprints. Three implementations pass all 1,118 controls and block all 5,047 specified violations in a 6,165-case offline suite.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedAgent_050.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=fIfFrQzvry
BibTex
@InProceedings{VohSid_AbsentByte_MICCAISAT2026,
author = { Vohra, Siddharth AND Ravikiran, Manikandan},
title = { { Absent-Byte Diagnoses: Auditing Structured Medical VLM Interfaces } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17263},
month = {pending},
page = {pending}
}