Abstract

Computer-aided medical image analysis is crucial for disease diagnosis and treatment planning. While vision-language models (VLMs) such as CLIP exhibit strong generalization ability, their direct application to medical imaging remains hindered by a substantial domain gap. Existing methods for bridging this gap, including prompt learning and unidirectional modality interaction, typically introduce domain knowledge into only one modality. However, such approaches fail to fully exploit CLIP’s inherent dual-modality structure and overlook the synergistic effect of bidirectional cross-modal interaction, resulting in persistent modality misalignment. In this paper, we propose NEARL (iNteracted quEry Adaptation with oRthogonaL Regularization), a novel parameter-efficient VLM framework for bidirectional cross-modal interaction. NEARL consists of two key components: (1) the Unified Synergy Embedding Transformer (USEformer), which dynamically generates compact cross-modal queries to facilitate interaction; and (2) the Orthogonal Cross-Attention Adapter (OCA), which decouples new knowledge into truly novel and incremental components through orthogonal regularization. This design reduces interference from incremental components, enabling more focused learning of novel information and improving modality interaction in VLMs. Notably, NEARL introduces only 1.46M learnable parameters. Extensive experiments on three medical imaging modalities demonstrate state-of-the-art performance (e.g., a 2.3% relative improvement on the pneumonia dataset), along with fast inference and low memory overhead, highlighting its effectiveness for real-world medical vision-language understanding.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0040_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{PenZel_NEARL_MICCAI2026,
        author = { Peng, Zelin AND Zhao, Yichen AND Huang, Yu AND Yang, Piao AND Tang, Feilong AND Xu, Zhengqin AND Yang, Xiaokang AND Shen, Wei},
        title = { { NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a bidirectional cross-modal interaction framework based on VLMs, named NERARL, aiming to address the domain gap when VLMs are directly applied to medical image analysis. The main contributions of this paper are “USEformer”, which is used to generate cross-modal queries, and “Orthogonal Cross-Attention Adapter”, which employs orthogonal techniques to decompose new knowledge and achieve the goal of accurately acquiring new information, with the aim of enhancing the modal interaction within VLMs.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1) The comparison with the CLIP backbone in terms of parameter count is interesting and suggests that NEARL achieves competitive inference throughput with limited parameter overhead. While this points toward potential resource efficiency, additional discussion on real-world deployment constraints would further strengthen the claim of clinical applicability. 2) The proposed adapter module appears to mitigate interference during incremental learning, as indicated by the ablation results. The design rationale is plausible, and the observed improvements in cross-modal interaction are encouraging. However, a deeper analysis of why orthogonality helps, via feature-space visualization or gradient dynamics, would provide more convincing mechanistic insight. 3) The evaluation spans multiple diseases and imaging modalities, which is a notable strength. Performance gains over prior methods are consistently reported, suggesting reasonable generalization. That said, dataset-specific performance variations and potential biases are not thoroughly examined; addressing these would improve confidence in the method’s robustness across heterogeneous clinical settings.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1) The baseline methods used in the comparative experiments are relatively outdated. It is recommended to include comparisons with more recent state-of-the-art approaches to better demonstrate the competitiveness of the proposed method. Alternatively, if incorporating newer baselines is not feasible, please provide a thorough discussion explaining why certain recent methods underperform compared to the classical baselines in this specific setting, thereby strengthening the validity and rigor of the experimental evaluation.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall, the paper is well-structured and clearly presents its experimental design and results, making the overall argument highly convincing. While there are minor shortcomings in the supporting evidence and the comparative experiments, these do not significantly detract from the core contributions. Given the methodological soundness, clear presentation, and strong empirical results, I recommend a Weak Accept, pending minor revisions to address the points raised above.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    After reading the rebuttal, I maintain my Weak Accept rating. The authors have reasonably addressed several concerns, especially regarding deployment efficiency, backbone choice, additional dataset results, and comparisons with recent baselines. The clarification of the orthogonality constraint also makes the motivation more convincing.



Review #2

  • Please describe the contribution of the paper

    The authors propose NEARL, a novel bidirectional cross-modality interaction framework based on vision-language models, comprising USEformer and OCA. USEformer generates learnable queries to enable bidirectional cross-modal enrichment, improving alignment between visual and textual features. OCA decouples truly novel knowledge from incremental updates, preventing interference with pre-trained features. Evaluations across three imaging modalities demonstrate that NEARL achieves superior performance.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.NEARL introduces a bidirectional framework that fully leverages the duality of vision-language models, improving alignment between visual and textual features, which addresses the limitations of prior unidirectional approaches. 2.The OCA module decouples truly novel knowledge from incremental updates using orthogonal regularization, preventing interference with pre-trained features and enabling focused acquisition of new domain-specific information. 3.NEARL achieves superior performance across multiple medical imaging modalities.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Questionable Motivation: While NEARL introduces USEformer and OCA to improve medical vision-language alignment, the overall novelty and practical significance are limited. The framework primarily relies on adapting a frozen CLIP model pretrained on natural image-text pairs with lightweight modules. Given the availability of numerous general-purpose and domain-specific medical foundation models, it is unclear why starting from a frozen natural-image CLIP is preferable to leveraging these pretrained medical models, which may already encode relevant medical visual-language representations. 2.Limited Evaluation Scope: Although NEARL is evaluated across three imaging modalities, experiments are limited to only three datasets, raising concerns about the generalizability to broader medical imaging datasets. Additional evaluation on more datasets could strengthen the evidence. 3.Unclear Task Definition: It is not explicitly stated whether the reported results correspond to zero-shot or other evaluation settings. Expanding the experiment to include multiple evaluation settings (e.g., few-shot, retrieval) would clarify the model’s applicability and robustness. 4.Visualization Comparisons: In Figure 2, the visualizations should be compared against representative bidirectional/unidirectional modality interaction baselines to better illustrate the contribution of the proposed modules to cross-modal alignment. 5.Parameter and Efficiency Analysis: While the authors claim that NEARL introduces only 1.46M additional parameters and achieves fast inference, it is unclear how this compares quantitatively to other baselines. More detailed analysis of parameter efficiency and runtime gains relative to existing methods is needed to support the claimed advantages.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    1.Questionable Motivation 2.Unclear Task Definition

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The response largely addresses my concerns, so I would like to recommend Accept.



Review #3

  • Please describe the contribution of the paper

    The main contributions of this paper are twofold. First, it proposes the Unified Synergy Embedding Transformer (USEformer), a lightweight module that enables fully bidirectional cross-modal interaction between visual and textual modalities by dynamically generating compact cross-modality queries to extract and fuse complementary information from both sides, achieving deep mutual alignment instead of the unidirectional interaction used in previous works. Second, it introduces the Orthogonal Cross-Attention Adapter (OCA), which applies Gram-Schmidt orthogonal regularization to decouple newly learned medical domain knowledge into truly novel domain-specific components and incremental components relative to the frozen pre-trained CLIP weights, preventing feature interference and helping the model focus on acquiring new medical knowledge efficiently. Besides these two core modules, the whole NEARL framework is designed to be parameter-efficient, only introducing 1.46M learnable parameters while achieving state-of-the-art performance across multiple medical imaging modalities, and it also provides strong empirical validation on three representative medical datasets covering X-ray, MRI, and OCT.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    One major strength is the novel bidirectional cross-modal interaction design, which differs from existing prompt learning or unidirectional modality adaptation methods that only optimize one modality or pass information in a single direction; USEformer builds real two-way communication between image and text branches, which better exploits the dual structure of CLIP and alleviates modality misalignment in the medical domain, making the cross-modal alignment more thorough and suitable for medical vision-language tasks. Another strength is the orthogonal regularization mechanism in OCA, which uses a simple yet effective Gram-Schmidt process to strictly separate adaptive features from pre-trained features, avoiding interference with the original generalization ability of CLIP and making the domain adaptation more focused and stable, and this orthogonal design is computationally efficient compared with other orthogonalization methods like Householder transformation, fitting clinical real-time needs. The paper also shows strong evaluation completeness: it tests NEARL on three distinct medical datasets with different imaging modalities and disease types, compares with a wide range of state-of-the-art methods including prompt learning, unidirectional adaptation, and even unimodal medical models, and achieves clear performance gains, such as a 2.3% relative accuracy increase on the pneumonia dataset. Additionally, the framework is highly parameter-efficient with only 1.46M extra parameters, delivers fast inference speed (500 fps on RTX 3090) and low memory overhead, which demonstrates practical clinical feasibility for real-time diagnostic screening. Comprehensive ablation studies and visual feature analysis further verify the effectiveness of each core module and explain the limitations of prior unimodal adaptation strategies, making the work solid and convincing.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The main weakness is that the core ideas of cross-modal interaction via learnable queries and adapter-based fine-tuning with regularization are not completely new in the vision-language field. Learnable query tokens for cross-modal fusion have been explored in many prior VLM adaptation works such as MaPLe and CoCoOp, and orthogonal regularization has also been used in some parameter-efficient adaptation methods to reduce feature conflict, so the overall paradigm of combining query-based interaction and orthogonal adapter follows existing technical routines without a radically new methodological foundation. Also, the experiments are limited to standard medical image classification tasks on three public datasets, without testing on more challenging medical vision-language tasks like image captioning, visual question answering, or lesion segmentation, which limits the demonstration of the framework’s generalization to broader medical VL understanding scenarios. Furthermore, the paper does not provide in-depth analysis on the sensitivity of key hyperparameters such as the number of query tokens, query dimension, and the number of USEformer layers, and there is no comparison with other recent orthogonal or bidirectional adaptation methods specifically designed for medical VLMs, making the novelty and superiority of the design less distinctive in the medical domain literature.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My overall recommendation is mainly based on several key positive factors that outweigh the minor weaknesses. First, the paper addresses an important practical problem—the domain gap between general VLMs and medical imaging—and provides a clear, effective, and clinically applicable solution with two well-designed complementary modules. Second, the technical design is reasonable and targeted: bidirectional interaction fixes the flaw of unidirectional adaptation, and orthogonal regularization solves feature interference during domain adaptation, both of which are well-motivated and clearly validated. Third, the empirical results are strong and consistent across multiple datasets and modalities, outperforming a large set of competitive baselines with very low parameter overhead and high inference efficiency, showing real practical value for computer-aided medical diagnosis. Fourth, the paper is well-organized, with complete experiments, ablation studies, and visual analysis to support every claim, making it technically solid and easy to follow. Although the core paradigm is not completely revolutionary, the concrete combination and optimization for medical vision-language understanding are innovative enough, and the performance and efficiency gains are significant for real clinical use. Therefore, the work makes a meaningful contribution to medical VLM adaptation and deserves acceptance.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank all reviewers for their comments. R1-Real-world constraints: NEARL is deployment-friendly with low inference memory (~1 GB) and exceptional data efficiency (only ~4k images for pneumonia adaptation). R1-Evidence of orthogonality: Prior works (e.g., Cogswell et al., ICLR 2016) suggest that reducing feature redundancy improves generalization. If adapter updates are not orthogonal to frozen features, they introduce overlap and conflict. Baselines show nontrivial angles (>5°), while OCA enforces near-perfect orthogonality (90° ± 0.02°), enabling non-redundant adaptation and generalization. R1-Performance variations: Improvement margins vary across datasets due to differences in imaging modality, disease characteristics, and task difficulty. For instance, the visual overlap between normal and pathological patterns in X-rays makes adaptation inherently harder. Despite this, NEARL consistently delivers positive gains, highlighting its robustness. R1- Baseline recency: When tested against recent methods, e.g., HiCroPL (ICCV 2025) and Spotlighter (EMNLP 2025), NEARL consistently performs best, achieving 94.7% v.s. 89.7% and 82.9%, respectively, further demonstrating its effectiveness. R2-Backbone choice: NEARL is backbone-agnostic and plug-and-play. We adopt natural-domain CLIP because prior medical adaptation works (e.g., ViP) use the same backbone for fair comparison. With a medical backbone (PubMedCLIP), NEARL reaches 94.0%, up from 62.5% (zero-shot), and surpasses baselines e.g., FATE (86.8%) and MaPLe (92.3%). R2-Evaluation scope: Our evaluation spans three widely used medical imaging modalities: X-ray, OCT, and MRI. On ISIC dermoscopy dataset, NEARL achieves 89.1% acc, outperforming previous methods e.g., FATE (84.5%) and MaPLe (85.8%), further supporting its effectiveness across diverse datasets. R2-Unclear task definition: Our experiments follow the standard PEFT setting used in prior medical adaptation works (e.g., ViP, XCoOp): all methods are tuned on the full training split with a frozen backbone, rather than zero-/few-shot evaluation, ensuring fair comparison. Few-shot or retrieval-based evaluation is beyond our current scope and left for future study. Thanks! R2-Visualization comparisons: Most baselines freeze the image encoder. While methods like MaPLe introduce trainable tokens, it does not enforce compatibility with the frozen space. T-SNE shows MaPLe features exhibit weaker separability. Due to the text-only rebuttal format, we cannot include visualization here. R2-Parameter/efficiency comparison: Existing methods use 8.19K–26.11M parameters at 470–540 fps. NEARL uses 1.46M parameters at a comparable 500 fps, yet achieves the best overall performance. R3-Differences from recent general and medical adaptation methods: First, NEARL uses learnable vectors as compact cross-modal queries, rather than prompt-based adaptation methods as in CoCoOp, MaPLe, or HiCroPL, enabling layer-wise retrieval of complementary information from the opposite modality. Second, the orthogonality constraint in NEARL is imposed directly in feature space, rather than in weight or gradient space as in O-LoRA and BOFA, thereby effectively reducing feature redundancy and improving representation generalization. Third, unlike bidirectional methods like BiPVL-Seg that merely rely on self-/cross-attention interaction, NEARL performs lightweight CLIP adaptation with compact queries, lightweight adapters, and explicit feature-space decoupling. R3-Limited to classification: We focus on medical image classification because it is the standard evaluation setting adopted by prior medical CLIP-based methods, e.g., ViP and XCoOp, enabling a fair comparison. Since NEARL is task-agnostic, we will explore it on other tasks in future work. Thanks! R3-Ablation: Sensitivity analysis on key hyperparameters shows minor performance changes across settings, generally within 2%, indicating that NEARL is robust and not highly sensitive to hyperparameter choices.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The principal concerns the authors must address in rebuttal are:

    Motivation for using a frozen natural-image CLIP rather than existing medical foundation models (MedCLIP, BiomedCLIP, PubMedCLIP) as the backbone (R2). This is the most consequential critique and directly affects the claimed contribution. Novelty positioning relative to learnable-query (MaPLe, CoCoOp) and orthogonal-adapter prior work (R3); the combination is reasonable but the delta needs sharper articulation. Baseline recency — comparisons skew toward older methods (R1); authors should justify selection or add recent medical VLM adapters. Evaluation scope and task setting — only three datasets, and zero-shot vs. few-shot vs. fine-tuned protocol is ambiguous (R2). Clarification is essential for interpreting the gains. Quantitative parameter/efficiency comparison against baselines, not just absolute numbers (R2). Mechanistic evidence for orthogonality — feature-space or gradient-level analysis would substantiate the design claim (R1).

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    All of the 3 reviewers agreed to accept this paper after rebuttal.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    All the reviewers agree the acceptance.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    All reviewers provided final acceptance scores.



back to top