Abstract
Dense medical image segmentation is usually learned end-to-end, making training expensive and data hungry, especially when strong Transformer foundation models are available but hard to adapt efficiently. Classic Hypercolumns concatenate intermediate CNN features at each pixel to obtain rich per-pixel representations, yet remain underexplored in medical imaging and are essentially absent for 3D volumes. We introduce HyperPatch, a patch-level Hypercolumn analogue for Transformer backbones. For each patch token, we extract representations from every Transformer block and concatenate them into a high dimensional vector, yielding a multi-level descriptor with local patch content and self-attention-induced global context. We couple HyperPatch with lightweight decoders that transform these vectors into dense 2D and 3D outputs, enabling pixel- and voxel-level predictions while keeping the backbone fixed. On 2D retinal vessel and endoscopy segmentation, HyperPatch consistently reaches a Dice score of 0.8 with only two to four training images, matching or surpassing state-of-the-art performance. We also provide feasibility results on 3D liver segmentation in abdominal CT. HyperPatch offers a simple and efficient strategy to repurpose pretrained Transformers for dense medical segmentation using a small trainable head and very limited training data. Code will be released upon acceptance.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/MLMI_079.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=AUZmfuefCk
BibTex
@InProceedings{SabAmi_HyperPatch_MICCAISAT2026,
author = { Sabour, Amirhossein AND Khokhar, Sahib AND Moradi, Mehdi},
title = { { HyperPatch: Hypercolumns for Transformer Backbones in Medical Segmentation } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17272},
month = {pending},
page = {pending}
}
