Abstract

Automatic landmarking of 3D facial scans typically relies on task-specific, data-intensive supervised models. We introduce FaceLandmarkAgent, a training-free multi-agent framework that connects a commercial vision-language model (VLM) with deterministic local geometric operators through a structured Responses API. The system follows a closed-loop planner–critic architecture: a planning agent selects camera viewpoints and invokes typed local Python tools to rotate and render the mesh, while structured 2D VLM proposals are lifted back to the 3D surface through ray–mesh intersections. A belief-state critic evaluates multi-view geometric dispersion, bilateral symmetry, midline ordering, surface validity, and execution cost, and requests targeted additional observations until a stopping criterion or the predefined budget is reached. We evaluated the framework on a 40-scan BU-3DFE holdout across nine soft-tissue landmarks using a common physician-annotated and expert-adjudicated reference standard. The full agent achieved a global mean radial error of 3.92 ± 2.45 mm, compared with 4.72 ± 2.68 mm after removing iterative re-querying (t=1) and 5.62 ± 3.12 mm for a heuristic single-view VLM baseline; the best supervised baseline achieved 2.42 ± 1.15 mm. These results demonstrate that closed-loop spatial–semantic planning and deterministic geometric grounding substantially improve zero-shot VLM landmarking, while supervised geometric models retain higher absolute accuracy on the evaluated benchmark.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MedAgent_047.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to Open Review

Open Review Page: https://openreview.net/forum?id=YYHeTO4gTq

BibTex

@InProceedings{GaoYao_FaceLandmarkAgent_MICCAISAT2026,
        author = { Gao, Yao AND Hou, Yiwen AND Hu, Jinyu AND Sun, Wenyue AND Li, Feng AND Wang, Yiwei AND Van Dessel, Jeroen AND Willaert, Robin AND Sun, Yi},
        title = { { FaceLandmarkAgent: A Training-Free Planner-Critic Agent for 3D Facial Landmarking via Spatial-Semantic Reasoning } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 17263},
        month = {pending},
        page = {pending}
}


back to top