Abstract
Many clinical questions about chest X-rays cannot be answered from images alone, as they depend on longitudinal context, including prior studies, earlier reports, diagnoses, and the timing of clinical events. Most medical vision-language models still process isolated image-question pairs. We present MedHyperGraph, a per-patient multimodal hypergraph for electronic health record (EHR)-grounded medical visual question answering (VQA). It integrates structured EHR events, prior-report entities, imaging studies, ontology concepts, and image regions into a single temporally indexed structure; evidence is then retrieved under explicit temporal constraints. On EHRXQA boolean questions ($n=1{,}900$), the retrieval baselines remain close to a majority-class predictor on the MedGemma backbones; MedHyperGraph improves on all three backbones, reaching $73.95\%$ accuracy ($95\%$ CI $[71.9,75.9]$), without task-specific training. On open-ended questions, it raises weighted F1 by $10.0$ points at 27B, driven by multi-study questions ($5.2\rightarrow43.4$). Ablations attribute the gain primarily to the structured EHR ($-9.1$), indicating that EHR-grounded VQA benefits from retrieval that preserves the temporal and $n$-ary structure of clinical evidence. Our code is available on GitHub: https://github.com/minaraafat21/MedHyperGraph.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/EMERGE_037.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=Gm7QeQBZxm
BibTex
@InProceedings{HeiMin_MedHyperGraph_MICCAISAT2026,
author = { Heinein, Mina AND Youssef, Mina AND Nashed, Kirellos AND Basha, Tamer AND Alam, Hasan Md Tusfiqur AND Selim, Abdulrahman Mohamed AND Bhatti, Omair Shahzad AND Sonntag, Daniel},
title = { { MedHyperGraph: EHR-Integrated Multimodal Hyperedges for Clinical VQA } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17266},
month = {pending},
page = {pending}
}
