Abstract
Scaling self-supervised foundation feature learning (SSL) to 3D medical imaging and dense prediction tasks is challenging due to the subtlety of small anatomical structures in large 3D volumes. Many codebook quantisation and masked image modelling concepts that excel in natural vision are difficult to replicate. In recent work substantial computational efforts, including complex network adaptations, student-teacher paradigms, and extremely resource-intensive pre-training, were necessary to obtain competitive SSL frameworks for 3D segmentation. Furthermore, many current approaches are prone to representation collapse and severe distribution shifts between the pre-training and inference. We address these challenges by proposing a symmetric red-black masking strategy combined with recent advances in finite scalar quantisation (FSQ) and local 3D neighbourhood attention transformers (NAT). The initial auto-encoding pre-training stage robustly maps each 888 non-overlapping patch to a discrete lattice of 1,000 integers without a learnable codebook or exponential moving average (EMA) updates. The subsequent masked latent modelling stage enforces multi-directional structural continuity by predicting the code of six orthogonal neighbours for every visible token in a symmetric red-black checkerboard pattern. A dual pass interleaved inference strategy for a six-block 3D NAT encoder is then applied to eliminate the pretrain-to-downstream distribution shift. Our experimental validation on 200 CT scans from AMOS22 demonstrates that even ViT-S backbones (frozen completely for downstream evaluation) achieve high segmentation performance. Evaluated with a restrictive, shallow strictly local two-layer fully connected network (FCN) for supervised probing, our approach yields a 54% Dice score for the pancreas, which is further increased to 62.4% when simply incorporating 6 spatial neighbours into the FCN probing layer. Our ablation demonstrate clear gains of the proposed red-black masking and 6-neighbour prediction (up to 7% points) and we also outperform the latest, much heavier, 3DINO framework. Our work demonstrates that powerful and dense 3D SSL features can be learned within hours and with minimal computational resources (<4 GByte VRAM). They do not require complex task-specific segmentation decoders or end-to-end fine-tuning.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/EMA_040.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: Not Available
BibTex
@InProceedings{HeiMat_Guess_MICCAISAT2026,
author = { Heinrich, Mattias P.},
title = { { Guess where the pancreas is: scalable 3D foundation features with Finite quantisation, neighbourhood attention and red-black masking } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17265},
month = {pending},
page = {pending}
}
