Abstract
Adapting a medical vision-language model (VLM) such as BiomedCLIP to a clinical service
is as much a stability problem as an accuracy problem: an update for a new imaging
modality can silently lose competence on modalities already handled and drift from the
trustworthy pretrained prior toward modality-specific shortcuts. Elastic weight
consolidation (EWC), the standard remedy, applies one fixed multiplier to an
unnormalized, accumulating Fisher, so the effective constraint depends on task loss
scale and on how many modalities have been consolidated, and the same hyper-parameter is
too weak under one arrival order and too strong under another. We show this is a
\emph{scale} effect and remove it by construction. CADRE keeps the backbone frozen and
adapts through low-rank adaptation (LoRA), regularized by an online, self-scaling
consolidation term over a sum-normalised Fisher plus an anchor-to-prior penalty on
embedding drift; consolidation mass is then bounded independently of the number of
modalities and the self-scaled penalty is invariant to rescaling. On a deliberately
dissimilar three-modality stress test (breast histopathology, ultrasound, chest
radiography) at $\approx$0.23\% of parameters, CADRE reduces forgetting roughly
sevenfold against the strongest LoRA-family baseline ($0.075!\rightarrow!0.011$;
paired $p{=}0.023$, $n{=}6$) and is the only method compared with positive backward
transfer.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026-sat/paper/SAFER_022.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to Open Review
Open Review Page: https://openreview.net/forum?id=kjDMBwujbW
BibTex
@InProceedings{JhaRis_CADRE_MICCAISAT2026,
author = { Jha, Rishabh AND Singh, Amrita AND Chudal, Prashanna},
title = { { CADRE: Stable, Parameter-Efficient Adaptation of Medical Vision-Language Models with Bounded Forgetting and Prior Drift } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026 Workshops and Challenges},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 17262},
month = {pending},
page = {pending}
}