Abstract
Vision-language models (VLMs) are increasingly proposed
for pathology image interpretation, yet existing benchmarks are curated
at a few academic centers, skew toward closed-form questions, and rarely
reveal how performance varies across patient populations. We introduce
PaiXBench, an open, expert-curated benchmark of 686 real histopathology
consultation cases drawn from the international iPath telepathology
network, spanning 16 countries across four world regions and 16 organ
systems. Each case pairs a representative image with structured clinical
context and a pathologist-curated reference diagnosis, and models answer
the diagnostic question in free text, mirroring real consultation practice
rather than multiple choice. Answers are scored by an LLM-as-judge validated
against pathologist grades (89.4% agreement, κ=0.79). Evaluating
seven VLMs ( ve open pathology/medical, two frontier generalist), no
model exceeds 49% accuracy, and under standardized single-image evaluation
pathology-specialized ne-tuning confers no advantagethe two
most specialized systems rank last, while frontier generalists lead a veway
statistical tie. Accuracy is non-monotonic in di culty, models fail
not only on hard cases but also on benign easy ndings, while remaining
comparable across the geographically dominant Asian and African
cohorts. We release PaiXBench to support more realistic and equitable
assessment of pathology VLMs across diverse clinical settings.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/COMPAYL_076.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/COMPAYL_076_supp.pdf
Link to Open Review
BibTex
@InProceedings{BauJen_PaiXBench_MICCAISAT2026,
author = { Baumann, Jens AND Aswolinskiy, Witali AND Pavlova, Marina AND Laskorunskyi, Roman AND Kindruk, Yulia AND Fritz, Peter AND Quick, Danny AND Aichmüller-Ratnaparkhe, Manasi AND Adam, Patrick},
title = { { PaiXBench: Benchmarking Vision-Language Models under Real-World Conditions } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17251},
month = {pending},
page = {pending}
}