Abstract
Agentic AI promises medical software that perceives, plans, and acts through domain tools rather than executing predefined functions, but every current system depends on a proprietary model behind a cloud API, conflicting with patient data governance and air-gapped workstations common in hospitals. In prior work we introduced a multi-agent architecture mapping natural language requests onto validated 3D Slicer CLI calls for a 23-tool dental imaging suite [9]; the agent is now deployed as 3D Slicer scripted modules, running inside the application. That system left open which retrieval strategy best feeds the routing agent as the tool repository grows, and how much accuracy a local, open-weight 8B-parameter pipeline sacrifices relative to a frontier model. We answer both on an expanded 250-query benchmark spanning register, verbosity, four languages, and incomplete requests, comparing four retrieval strategies (BM25, dense bi-encoder, hybrid RRF, cross-encoder reranking) feeding a schema-constrained Qwen3-8B router and extractor. Retrieval shows that top-3 recall, not top-1 precision, drives routing accuracy, and constrained decoding makes every output valid by construction. Running on-device at zero marginal cost, the pipeline reaches 94.4% exact-match routing and 81.2% parameter recovery, retaining 94.4% and 93.3% respectively of an unaided Claude Sonnet 4.6 reference. The residual gap concentrates in the strict all-fields-correct metric: a clinician would need to check over half of the local pipeline’s commands before running them, against four in ten from the frontier model, a difference in oversight burden rather than tool selection or parameter accuracy alone.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedAgent_008.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=5tMpwlwa9n
BibTex
@InProceedings{DumPau_AFully_MICCAISAT2026,
author = { Dumont, Paul AND Buisson, Alexandre AND Cevidanes, Lucia H. AND Prieto, Juan Carlos AND Castillo, Aron Aliaga-Del AND Bianchi, Jonas AND Teixeira, Rodrigo AND Pieper, Steve AND Li, Tengfei AND Zhu, Hongtu AND Mattos, Claudia Trindade},
title = { { A Fully Local Agentic LLM Architecture for Tool Invocation in Medical Imaging } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17263},
month = {pending},
page = {pending}
}