Abstract
Interactive segmentation plays a crucial role in accelerat-
ing the annotation, particularly in domains requiring specialized exper-
tise such as nuclear medicine. For example, annotating lesions in whole-
body Positron Emission Tomography (PET) images can require over an
hour per volume. While previous works evaluate interactive segmenta-
tion models through either real user studies or simulated annotators,
both approaches present challenges. Real user studies are expensive and
often limited in scale, while simulated annotators, also known as robot
users, tend to overestimate model performance due to their idealized na-
ture. To address these limitations, we introduce four evaluation metrics
that quantify the user shift between real and simulated annotators. In
an initial user study involving four annotators, we assess existing robot
users using our proposed metrics and find that robot users significantly
deviate in performance and annotation behavior compared to real anno-
tators. Based on these findings, we propose a more realistic robot user
that reduces the user shift by incorporating human factors such as click
variation and inter-annotator disagreement. We validate our robot user
in a second user study, involving four other annotators, and show it con-
sistently reduces the simulated-to-real user shift compared to traditional
robot users. By employing our robot user, we can conduct more large-
scale and cost-efficient evaluations of interactive segmentation models,
while preserving the fidelity of real user studies. Our implementation is
based on MONAI Label and will be made publicly available.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/ADSMI2024_086.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: https://papers.miccai.org/miccai-2026-sat/supp/ADSMI2024_086_supp.pdf
Link to Open Review
Open Review Page: Not Available
BibTex
@InProceedings{MarZdr_Rethinking_MICCAISAT2026,
author = { Marinov, Zdravko AND Kim, Moon AND Kleesiek, Jens AND Stiefelhagen, Rainer},
title = { { Rethinking Annotator Simulation: Realistic Evaluation of Whole-Body PET Lesion Interactive Segmentation Methods } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17276},
month = {pending},
page = {pending}
}