List of Papers Browse by Subject Areas Author List
Abstract
Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp’s 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5065_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{ZenJia_KMaT_MICCAI2026,
author = { Zeng, Jiajun AND Albarqouni, Shadi},
title = { { K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a novel cross-modal knowledge transfer framework, K-MaT, which achieves semantic alignment by decomposing prompts, performing knowledge anchoring, and integrating Gromov–Wasserstein fusion. The overall idea is clear, and the experiments cover multiple tasks, demonstrating a certain degree of innovation and research value. However, some details in the methodology description and experimental design still need further clarification and improvement. It is recommended that the manuscript be revised before reconsideration for publication.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(1) The proposed K-MaT in this paper enables zero-shot transfer from high-end imaging modalities (CT/MRI) to low-end modalities (X-ray/ultrasound), addressing issues such as catastrophic forgetting. This represents a highly practical and important scenario in medical imaging. (2) K-MaT adopts a decoupled architecture consisting of MSC and CSC, which helps reduce interference between different modalities. (3) The study compares multiple baseline models and conducts ablation experiments, with the results presented in comprehensive tables.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
(1) The paper mentions that the clinical text descriptions are generated using LLMs. Is this claim accurate? Have these descriptions been verified or validated by medical professionals? (2) How should the hyperparameter α in the FGW (Fused Gromov–Wasserstein) formula be chosen appropriately? (3) Lack of a sensitivity study of parameter λ. (4) Relying solely on semantic alignment, without aligning the high- and low-end visual features, may result in unstable performance across different scenarios. (5) The paper only explores the classification task, which is relatively limited in scope. (6) The cross-modal alignment of the language-image modalities has been widely discussed; the authors should discuss or compare the alignment strategy with related works such as [1][2].
[1] LICO: explainable models with language-image consistency, NeurIPS 2023. [2] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models, ICLR 2023.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Clear motivation of the manifold transport. Comprehensive experiments on several datasets.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper introduces K-MaT, a novel prompt-learning framework for zero-shot asymmetric cross-modal transfer in medical vision-language models. The method enables knowledge transfer from high-end imaging modalities (e.g., CT, mammography) to low-end modalities (X-ray, ultrasound) without requiring target-domain training data.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Knowledge-anchored prompt learning: The method incorporates LLM-generated clinical textual descriptions as semantic anchors to regularize learned prompts and mitigate modality-specific overfitting and catastrophic forgetting. 2.Factorized prompt design: The framework separates class-specific and modality-specific context tokens, allowing more flexible and robust prompt adaptation. 3.Cross-modal manifold alignment via Fused Gromov-Wasserstein (FGW) A key technical contribution is the use of FGW optimal transport to align the pairwise relational structure of low-end prompt embeddings to that of high-end embeddings.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Weak baseline coverage for cross-modal alignment: The comparison is restricted to prompt-learning methods. The paper does not include feature-level alignment or multimodal adaptation approaches. 2.Insufficient theoretical justification for FGW alignment While FGW is correctly formulated, the paper does not clearly explain why preserving pairwise relational structure leads to improved semantic transfer across modalities, beyond empirical observation. 3.Performance on low-end modalities remains limited Despite improvements, absolute performance on target modalities remains relatively low (~39% accuracy on average), indicating limited practical usability.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall, the paper presents a solid and technically sound method with consistent but incremental improvements, making it slightly above the acceptance threshold. The final decision could reasonably depend on rebuttal, particularly regarding baseline comparisons, justification of FGW alignment, and clarification of the zero-shot setting.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper proposes K-MaT, a zero‑shot prompt learning framework for asymmetric cross‑modal transfer in medical imaging. By factorizing prompts, anchoring them semantically, and aligning manifolds via FGW optimal transport, the method aligns the low‑end modal prompt manifold to a high‑end representation space with visual support. The authors claim that K‑MaT effectively mitigates catastrophic forgetting.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
(i) The paper introduces a spatial anchoring constraint to alleviate catastrophic forgetting. The idea of constraining learnable prompts from drifting toward, e.g., “modality‑specific statistical shortcuts” via semantic anchors is meaningful. (ii)The method uses semantic anchoring to prevent prompts from shifting toward modality‑specific appearances, and aligns the low‑end prompt manifold to the high‑end manifold via FGW to preserve shared diagnostic semantics and discriminative structure, achieving zero‑shot cross‑modal transfer without requiring low‑end training images.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
(i) The authors state in the contribution that “spatial anchoring constraint mitigates catastrophic forgetting” and mention in the ablation study that “Introducing L_{anc} … improving the H accuracy to 41.94% (row 5).” However, row 5 in Table 3 with H accuracy 41.94% corresponds to CSC+MSC+L_{fgw}, not CSC+MSC+L_{anc}. From Table 3, comparing row 3 (CSC+MSC) and row 4 (CSC+MSC+L_{anc}) would be more reasonable, but adding L_{anc} actually decreases H‑accuracy from 37.95% to 35.15% (row 4). Therefore, the ablation experiments do not adequately explain whether the spatial anchoring constraint effectively mitigates catastrophic forgetting. (ii) The interpretability of the FGW ablation experiment regarding alleviating collapse is weak, as it only shows a single task and insufficient evidence. (iii) The final performance of the method likely depends heavily on the specific high‑low modality pair. The authors should further analyse the sensitivity of the method to different modality relationships. (iv) The benchmark task settings in the experiments are still insufficient and the scenarios are not comprehensive. Moreover, most of the compared methods are not specifically designed for asymmetric high‑to‑low cross‑modal transfer; even so, K‑MaT does not consistently outperform on the target low‑end modality. Therefore, the average H being the best is not sufficient to fully demonstrate the method’s overall effectiveness in high‑to‑low transfer scenarios. (v) The paper contains numerous inconsistencies and errors (see item 12 for details), and the interpretability of the method modules is insufficient in the cross‑modal generalisation and ablation experiments.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
(i) Section 3.1 Experimental Settings Datasets and Tasks – repeated punctuation at the end: “ .. ” (ii) Table 1.“Dataset statistics for the five cross‑modal tasks.” The title says five tasks, but the table shows four tasks. (iii) “Best ACC/F1 results are highlighted in bold/underline respectively.” This statement is problematic. From the table, for the same metric, bold/underline indicate the best and second best. (iv) In the ablation study, the authors write “Introducing L_{anc} … improving the H accuracy to 41.94% (row 5).” However, row 5 with H accuracy 41.94 corresponds to CSC+MSC+L_{fgw}, not CSC+MSC+L_{anc}. (v) The authors are advised to re‑check the entire manuscript.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(2) Reject — should be rejected, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although the ideas of semantic anchoring and manifold alignment have some novelty, the current experimental design is still incomplete, the writing is confusing, and the evidence is insufficient. In particular, there is a factual error in the attribution of the effect of L_{anc} in the ablation study, with a clear contradiction between the text and the table. Combined with insufficient comparison experiments, missing reproducibility information, and frequent writing errors, I cannot recommend it.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We sincerely thank the reviewers and the meta-reviewer for their constructive feedback. We are encouraged that the relevance of zero-shot asymmetric high-to-low modality transfer, factorized prompt learning, and FGW-based manifold alignment was recognized. We also appreciate the comments that helped us identify where the manuscript should be clarified.
Ablation inconsistency (R3, Meta). We acknowledge that the ablation discussion contains an inaccurate attribution. In the sentence “Introducing L_anc … improving the H accuracy to 41.94%…” (page 7), the 41.94% result should be attributed to CSC+MSC+L_fgw rather than CSC+MSC+L_anc. The corrected interpretation is that L_fgw is the main contributor to the harmonic-mean improvement, improving H Acc to 41.94%, while the full model further improves it to 44.13%. Thus, L_anc alone is not sufficient, but is beneficial when combined with FGW in the full model. We will also revise broader statements to clarify that the mitigation comes from the combination of semantic anchoring and FGW-based structural transport, rather than anchoring alone.
Zero-shot setting and baselines (R1, R2, Meta). Our setting is a strict target-free transfer: during training, we use only high-end visual images and do not access low-end visual images, even without labels. Therefore, methods that align high- and low-end visual features usually require unavailable low-end visual embeddings. Comparing such methods would change the problem from target-free transfer to target-dependent adaptation. We will clarify this distinction and discuss LICO and PLOT as related vision-language/optimal-transport alignment methods, while emphasizing that they are not designed for this setting. We are aware that incorporating low-end visual embeddings is an important future extension, where prompt-space structural transport and visual-feature alignment can be jointly studied.
FGW motivation and hyperparameters (R1–R3). FGW is used to transfer inter-class relational geometry, not merely to match individual text embeddings. Since high-end prompts are visually grounded by supervised high-end images, while low-end prompts have no visual supervision, preserving the high-end pairwise class structure helps transfer class-discriminative information to the low-end prompt manifold. For the FGW trade-off α, we use a relatively small value to prioritize the direct cross-modal feature cost between high- and low-end prompt embeddings, while retaining the structural term as an auxiliary regularizer. This is important because the two imaging modalities can have substantially different visual characteristics; overemphasizing intra-modal inter-class geometry may impose an overly strong and potentially inaccurate structural prior on the low-end modality. For λ_anc and λ_fgw, we select them only using the high-end validation set to preserve the zero-shot low-end protocol. A more comprehensive sensitivity analysis of these regularization weights will be part of future work.
Regarding LLM-generated descriptions (R1), we acknowledge that they were not formally verified by clinicians. Our intention is not to use them as diagnostic labels, but as frozen class-level textual anchors to regularize learnable prompts. This follows prior VLM prompt-learning works, which use generated visual descriptions to reduce reliance on manually crafted expert prompts. The descriptions, therefore, keep learnable prompts within a meaningful linguistic-semantic space and reduce overfitting to unconstrained prompt directions.
Finally, we will fix minor comments and typos in our camera-ready version. We thank the reviewers again and believe these clarifications strengthen the validity and positioning of the paper.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This submission presents a relevant and technically coherent contribution on zero-shot asymmetric cross-modal transfer for medical vision-language models, but the current evidence places it in the borderline range rather than clearly above threshold. The strengths of the paper are the importance of the problem setting, the clear use of a frozen biomedical VLM with factorized prompts, semantic anchoring, and FGW-based manifold alignment, and the competitive results against the included prompt-learning baselines. In particular, the manuscript evaluates four medically meaningful high-to-low modality-transfer tasks and reports the strongest average harmonic-mean performance among the compared methods, which is an appropriate metric for a setting that aims to preserve source performance while improving target generalization. The decisive concerns are more specific. First, after checking the manuscript, one reviewer’s concern about the ablation study is correct: the text attributes the 41.94% harmonic-mean result to (L_anc), but Table 3 shows that this row corresponds to (L_fgw), while the row adding (L_anc) alone does not improve harmonic-mean accuracy. This weakens the current presentation of the anchoring component and must be clarified. Second, the paper compares only against prompt-learning baselines, so the empirical validation is narrower than ideal for a methodology paper. Third, the manuscript itself acknowledges that absolute low-end performance remains limited and that purely text-anchored alignment may not fully bridge severe visual discrepancies. At the same time, some negative review points are overstated: the manuscript does provide substantial implementation detail, and the empirical evidence is not invalidated simply because K-MaT is not best on every individual target task. Accordingly, I assign limited weight to those overstated criticisms in the overall assessment.