Abstract
We present a multimodal large language model that reproduces a pathologist’s diagnostic workflow directly from a whole-slide image (WSI). Instead of emitting a single label or a free-form report, the model traverses a tree of clinical question-answer steps that terminates in a pathology report. An attention-based multiple-instance-learning (ABMIL) slide encoder is coupled to a quantized medical LLM through a lightweight visual projector. The central design choice is that the model learns only to answer, while the follow-up questions are generated deterministically by an engine that consults an organ- and procedure-specific decision schema derived automatically from the training annotations; thus, branching logic is never learned, and training mirrors inference. We pre-trained the slide encoder to predict the organ, procedure, abnormality, and severity, and co-trained slide-level reasoning with region-of-interest grounding in a single run. On two held-out validation sets, the system reaches a Workflow Ranking Score (WRS) of 0.846 on a random split and 0.796 on a harder stratified split. Removing the schema costs 0.130 WRS at the same backbone scale, which we attribute to the 4-8B model being unable to learn question generation and answering simultaneously without overfitting the former. This makes the schema a way of spending a constrained capacity budget, rather than claiming that structure should never be learned.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/REG2026_001.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/REG2026_001_supp.pdf
Link to Open Review
Open Review Page: https://openreview.net/forum?id=M9zV9wXVCq
BibTex
@InProceedings{KimJih_SchemaGuided_MICCAISAT2026,
author = { Kim, Jihwan AND Jeong, Ji-Hoon AND Choi, Yoon-La},
title = { { Schema-Guided Tree-Structured Reasoning for Whole-Slide Pathology Workflows } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17251},
month = {pending},
page = {pending}
}