Abstract
Hierarchical skin disease classification must recognize broad disease categories while preserving distinctions among clinically similar subclasses. Parent-level contrastive supervision can work against this goal: sibling diagnoses sharing the same parent are treated as positives, creating a coarse–fine conflict that limits fine-grained discrimination. We introduce a taxonomy-guided vision–language framework that uses language not merely as an input or output modality, but as structured supervision for learning a visual classifier. Multiple concept-derived captions constructed from lesion morphology, anatomical location, symptoms, and disease descriptors provide clinically grounded multi-positive supervision, while same-parent/different-subclass pairs receive increased negative importance according to the disease taxonomy. The resulting representation is coupled with a gallery-based prototype classifier for hierarchy-constrained prediction and retrieval of clinically grounded caption exemplars. Because classes are represented by textual prototypes rather than a fixed output head, the frozen encoders can also operate over a disjoint disease vocabulary through gallery replacement. Taxonomy-guided optimization raises Sub-Class macro-F1 from 14.05 to 32.01, while the complete framework reaches 54.21 and substantially outperforms a frozen-CLIP linear probe. On an independent dataset with non-overlapping labels, it also outperforms vanilla CLIP when both use the same labelled target gallery and prototype construction. These results suggest that clinical language and disease taxonomy provide complementary supervision for fine-grained visual representation learning and flexible hierarchical classification.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/ISIC_024.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=X834Dl527F&nesting=2&sort=date-desc
BibTex
@InProceedings{DeeAnj_TaxonomyGuided_MICCAISAT2026,
author = { Deepu, Anjali AND Dey, Swarnava},
title = { { Taxonomy-Guided Vision–Language Representation Learning for Hierarchical Skin Disease Classification } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17271},
month = {pending},
page = {pending}
}
