Abstract

The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic knowledge contained in clinical reports. Furthermore, effectively leveraging these records is hindered by a fundamental modality gap: structured anatomical descriptions are not naturally mapped to specific frames within the high-redundancy, uncurated visual streams. In this paper, we present EndoVLM, a novel vision-language FM pre-trained on over 348K endoscopic examinations, each pairing a clinical report with its corresponding image collection. An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention, efficiently aggregating semantically salient frames into anatomy-specific visual representations across redundant image-sets. Next, a Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets, bridging the gap from global patient-level matching to fine-grained localized alignment. Finally, a Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation. Extensive experiments across various downstream tasks demonstrate that EndoVLM outperforms existing foundation models and remains competitive with task-specific methods. Remarkably, EndoVLM also exhibits robust zero-shot generalization capabilities, highlighting its potential for broader clinical application. Codes are available at https://github.com/Scatteredrain/EndoVLM.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0285_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/Scatteredrain/EndoVLM

Link to the Dataset(s)

LIMUC dataset: https://github.com/GorkemP/labeled-images-for-ulcerative-colitis PolypDiag dataset: https://github.com/tianyu0207/weakly-polyp CVC-12k dataset: https://polyp.grand-challenge.org/Databases/ Hyper-Kvasir dataset: https://datasets.simula.no/hyper-kvasir/ Kvasir-SEG dataset: https://github.com/DebeshJha/Kvasir-SEG ClinicDB / ColonDB dataset: https://pages.cvc.uab.es/CVC-Colon/index.php/databases/ ETIS dataset: https://www.kaggle.com/datasets/nguyenvoquocduong/etis-laribpolypdb

BibTex

@InProceedings{YiZhe_EndoVLM_MICCAI2026,
        author = { Yi, Zhenyu AND Xu, Jianwei AND Hu, Yue AND Qiu, Zhongwei AND Li, Sijing AND Huang, Liang AND Lv, Bin AND Zhang, Ling AND Xia, Yingda},
        title = { { EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper
    • The paper presents a novel vision-language foundation model pre-trained on a large-scale dataset of 348K endoscopic examinations.
    • Extensive experiments are conducted across various downstream tasks, showing competitive or improved performance.
    • The method introduces three main components to effectively handle unordered image sets and raw clinical reports.
  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The proposed VLM is pre-trained on over 348K endoscopic examinations, including both image and clinical reports.
    • The paper addresses the semantic gap between redundant visual frames and structured clinical narratives, which is an important challenge in medical vision-language model design.
    • The framework is evaluated across five diverse downstream tasks.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The choice of encoders (ViT-B/16 for vision and PubMedBERT for language) is not sufficiently justified. It would be helpful to understand whether alternative architectures were explored and how sensitive the method is to this choice.
    • In Table 1, the task-specific baselines perform comparably to the proposed method. This raises questions about the approach’s actual benefit, and a statistical significance analysis would strengthen the claims.
    • The difference between “seen” and “unseen” datasets in Table 1 is not clearly explained. It is unclear whether the seen datasets were part of pretraining.
    • The paper lacks qualitative results on downstream tasks, which would provide better insights into model behavior and interpretability.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
    • The paper presents a large-scale vision-language framework trained on 348K endoscopic examinations, which is a strong contribution.
    • The performance gains over task-specific baselines are marginal, and no statistical significance analysis is provided to support the improvements.
    • The choice of encoders is not sufficiently justified, and it is unclear how sensitive the method is to these design decisions.
  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper presents EndoVLM, a novel vision-language foundation model pre-trained on a large-scale corpus of more than 348,000 endoscopic examinations. The method consists of three main components. First, SC-MAE performs image-level pre-training from a reconstruction perspective, encouraging the model to capture low-level visual structure and appearance cues from selected semantic-rich frames. Second, AGSP uses anatomy-aware textual features, extracted from clinically structured reports, to attend to and aggregate the most relevant image features corresponding to each anatomical region. Third, PSAA introduces a progressive contrastive alignment strategy at both the patient level and the anatomy level, aiming to bridge coarse global matching and finer-grained semantic correspondence between visual and textual representations.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper proposes an interesting fine-grained alignment strategy by introducing anatomy-aware soft-label supervision. In particular, the method does not treat all non-matching pairs as equally negative; instead, it leverages medical taxonomy together with textual semantic similarity to define which pairs should be considered closer, and then uses this target distribution to supervise visual-text alignment. 2.The experiments are extensive and generally well organized. The authors evaluate the model across several downstream tasks, include zero-shot transfer results, and provide ablation studies for the major components, which makes the empirical section reasonably thorough.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The paper is often unclear in writing and presentation, which substantially hurts readability. The high-level motivation is not always presented in a fully consistent way. In the introduction, the paper emphasizes that existing foundation models for endoscopy are mainly uni-modal, yet several of the compared baselines are themselves multimodal vision-language models. This makes the positioning of the work less precise than it could be. The notation in Section 2 is overloaded and sometimes confusing. For example, the symbol s is reused for semantically different quantities, and the paper switches rather abruptly between different indexing schemes and variable groups, which makes the derivation difficult to follow. In addition, some symbols such as the image/frame notation and the selected index sets are not introduced as cleanly as they should be. The model figure is also not sufficiently self-explanatory. It is difficult to understand the full pipeline directly from Fig. 1 without repeatedly referring back to the text, and some visual/layout details further reduce readability. 2.The design of the fine-grained soft target is interesting, but its underlying assumption is not fully justified. The proposed soft label in PSAA is built from anatomy labels, pathology status, and text-text semantic similarity, and is then used to supervise visual-text similarity distributions. This implicitly assumes that semantic proximity between text descriptions within the same anatomy/pathology space is a reasonable proxy for the corresponding visual similarity. However, this assumption is not sufficiently discussed or validated. In medical imaging, it is entirely possible that two reports are written in similarly templated language while the actual lesion appearance differs substantially. In such cases, text similarity may not faithfully reflect visual similarity. Relatedly, the current formulation uses t-t similarity to construct the target distribution, while the predicted distribution is based on v-t similarity. What if v-v similarity be applied? 3.The zero-shot anatomy recognition results appear uneven and raise concerns about robustness across anatomical categories. While the paper highlights near-perfect performance for some anatomy recognition tasks, the performance on ileum is much lower, and appears notably worse than BiomedCLIP in Fig. 2.This large gap across anatomical classes raises concerns about whether the learned representation generalizes uniformly across anatomies, or is biased toward categories that are visually more distinctive or more closely aligned with the pre-training supervision. 4.Some design choices are under-explained, especially the selection of Top-K. In AGSP, the model selects the Top-K most similar frames for each textual proposition, with K=3.However, the paper does not clearly explain how this value was chosen, whether it was tuned on validation data, or how sensitive the method is to this hyperparameter. Since AGSP is a core module, this design choice deserves stronger justification or at least a small ablation.

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    The work is based on a very large-scale collection of human clinical endoscopy data from hospitals, so the paper should clearly state the relevant ethics approval.

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My recommendation is mainly influenced by two concerns: unclear presentation and insufficiently justified methodology. The paper is often hard to follow because of inconsistent motivation, overloaded notation, and figures that do not clearly explain the model. More importantly, one of the core assumptions of the method—that anatomy-aware text similarity can serve as a suitable target for supervising visual-text alignment—is interesting but not convincingly validated.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I agree to accept this paper as the author addressed my concerns in the rebuttal section.



Review #3

  • Please describe the contribution of the paper

    The paper proposes EndoVLM, a vision-language foundation model tailored for gastrointestinal endoscopy, addressing the mismatch between structured clinical reports and unordered, redundant image sets. It introduces an anatomy-guided sparse pooling mechanism to retrieve semantically relevant frames from large image collections using textual cues. A progressive semantic-aware alignment strategy is designed to bridge global patient-level matching and fine-grained anatomical and pathological alignment. Additionally, a semantic-concentrated masked autoencoder is applied to selected informative frames to enhance representation learning. Extensive experiments on multiple downstream tasks demonstrate strong performance and generalization, including notable zero-shot capabilities.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.This work proposes a novel formulation of endoscopy vision-language pre-training by explicitly modeling the report-to-image-set alignment problem, rather than the standard one-to-one image-text paradigm. This is particularly interesting because it better reflects real clinical workflows, where a single structured report corresponds to many unordered and redundant frames, and thus addresses a practical limitation of existing VLM approaches.

    2.This work proposes an anatomy-guided sparse pooling mechanism that leverages fine-grained textual descriptions to retrieve semantically relevant frames from large image sets. This design is well aligned with the redundancy in endoscopic data and provides an effective way to bridge weak supervision from reports to localized visual evidence.

    3.This work demonstrates strong empirical performance through a comprehensive evaluation across multiple downstream tasks, including classification, segmentation, and zero-shot transfer. The consistent improvements over both general foundation models and endoscopy-specific methods, as well as the demonstrated generalization to unseen datasets, support the effectiveness and robustness of the proposed framework.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Lack of clear methodological novelty beyond integration of existing components. While the overall framework is well designed, key elements such as sparse attention for frame selection, contrastive alignment, and MAE-based reconstruction have been explored in prior works (e.g., CLIP [17], MAE [9], SimCROP [28], fVLM [19]). The paper would benefit from a clearer articulation of what is fundamentally new versus what is adapted, especially in comparison to recent endoscopy foundation models such as EndoFM [30] and EndoFM-LV [31].

    2.Lack of validation and analysis for the Qwen3-based report parsing pipeline. The entire framework relies on extracting anatomy and pathology triplets from reports, yet there is no quantitative evaluation of parsing accuracy or robustness to noise. It remains unclear how sensitive the model is to incorrect or ambiguous extractions, and whether the reported gains partly depend on implicit supervision quality from the LLM. 3.Lack of sufficient justification and analysis for the soft target design in Eq. (4). The use of cosine similarity between textual embeddings to weight abnormal cases appears heuristic, and the asymmetry between normal and abnormal alignment is not well explained. It would be important to understand whether this formulation is critical or whether simpler alternatives (e.g., hard labels or uniform weighting) would yield similar performance.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper tackles a realistic and important problem in endoscopy by modeling report-to-image-set alignment, which is a meaningful step beyond standard VLM settings. The framework is well designed and achieves consistent improvements across multiple tasks with large-scale pretraining.

    However, the methodological novelty is moderate, and several aspects such as LLM-based parsing reliability and key design choices are not sufficiently analyzed. Overall, the work is solid with strong empirical results, but still somewhat incremental in terms of core innovation.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers for recognizing the “strong contribution” (R1), “large-scale corpus” (R2), and “novel formulation” (R3). We address key concerns below.

@R2-Q1, R3-Q1 Positioning/novelty. First, regarding baselines: current endoscopy FMs (e.g., EndoFM, EndoMamba) remain largely visual self-supervised methods, and the evaluated VLM baselines are general/biomedical models transferred to our corpus, not prior endoscopy-domain VLMs. Our goal is therefore to build a dedicated endoscopy-specific vision-language foundation model. Rather than simply combining existing components, we introduce targeted adaptations for the endoscopic setting of mapping one report to many redundant frames: PSAA progressively unifies anatomy/pathology supervision in a shared latent space via soft targets, AGSP provides the annotation-free local visual units required for this alignment, and SC-MAE reuses AGSP-selected semantic-rich frames for additional visual supervision. This distinguishes EndoVLM from prior uni-modal endoscopy FMs and generic medical VLP settings.

@R2-Q2, R3-Q3 PSAA soft target. We clarify that PSAA uses text similarity not as a proxy for visual similarity, but as a soft relational prior to avoid treating semantically compatible texts as equally hard negatives when shaping the visual-text similarity distribution: if t_k is semantically close to paired text t_u under the same anatomy/pathology constraints, penalizing (v_u,t_k) as a hard negative would introduce false negatives. Moreover, we use text-text rather than visual-visual similarity because endoscopic visual similarity is unstable and easily confounded by acquisition factors, while the PubMedBERT-initialized text space provides a more stable semantic anchor. Clinically, PSAA is designed to impose anatomy- and pathology-aware structure more relevant to downstream diagnosis, rather than to learn a purely visual fine-grained similarity space. This also motivates the asymmetric design: normal samples within the same anatomy share stable healthy patterns, whereas abnormal samples are more heterogeneous, making hard labels overly restrictive. Consistently, a preliminary LIMUC comparison showed that hard-label, same-anatomy uniform, and same-anatomy+pathology uniform targets yielded AUCs of 93.8/94.1/94.2, all below full PSAA (94.5).

@R2-Q3 Zero-shot imbalance on ileum. The lower ileum result is better understood as a category-specific limitation: colonoscopy reports often mention ileum procedurally, e.g., inserting into the terminal ileum, rather than as a consistently described anatomical region, making supervision rarer and less uniformly defined. We will clarify this limitation.

@R2-Q4 Pooling K choices. As a preliminary sensitivity check for AGSP, LIMUC AUCs at K={1,3,5,7,10,20,all-attn} are 94.2/94.5/94.0/94.1/94.2/93.9/94.1, supporting K=3 while suggesting limited sensitivity.

@R2 Presentation/ethics. We will streamline overloaded notations, clearly define index sets, and make Fig. 1 self-contained. Pretraining data were de-identified with ethics approval; details will be disclosed upon acceptance.

@R1 Experimental clarification. We chose ViT-B/16 as a standard backbone comparable to baselines, and PubMedBERT for stable biomedical semantics. For segmentation, our claim is transferability: EndoVLM features with a minimal decoder (1x Conv) achieve competitive performance against engineered task-specific models. “Seen/unseen” only denotes downstream fine-tuning visibility; all Table 1 datasets are disjoint from our private pretraining corpus. Qualitative samples will be added to our repository.

@R3-Q2 Qwen3 parsing pipeline. This is a constrained extraction task over predefined fields, not open-ended generation. As part of dataset quality control, three experienced endoscopists audited 1000 random LLM-extracted sub-reports, yielding 99.8%/99.5% accuracy for anatomy/pathology states. We will clarify the parsing details and note residual extraction noise as a limitation.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Reviewers mention that the contribution in itself is relevant and strong, although with limited novelty. However, the presentation of the work could be considerably improved and the analysis are limited both in scope and depth. So this is really a borderline case.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Two of three reviewers initially proposed to accept the paper, but did not engage in the rebuttal. One reviewer proposed a weak reject pre-rebuttal but indicated that her/his concerns were addressed in the rebuttal. Although reviewers indicated that the novelty was limited and technical choices could have been better motivated, the strong empirical contribution motivates the suggestion to accept this paper.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The contribution of this paper to medical vision-language research is appreciable. Most of the concerns raised by the reviewers were addressed in the rebuttal document. Therefore, I believe this paper is worthy of acceptance.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper addresses an important problem in endoscopy VLM pre-training, and reviewers recognized meaningful contributions. Although the novelty is moderate and the presentation/analysis had some weaknesses, the rebuttal reasonably clarified key concerns. Overall, the work is solid and empirically strong.



back to top