Abstract

Multimodal biomedical Vision-Language Models (VLMs) confront a core dilemma in Continual Learning (CL): fine-grained feature confusion and blurred decision boundaries arising from the distinct data distribution characteristics of medical imagery. To address this challenge, we propose a novel Misdiagnosis-Aware Structured Memory framework that explicitly embeds misdiagnosis lessons into the retrieval topology, enabling the model to perform diagnostics akin to an experienced clinician. Specifically, we build a hierarchical Pitfall Graph that connects medical entities to their definitions, common mimics, and exclusion criteria, and integrate it into a Retrieval-Augmented Generation (RAG) system to guide fine-tuning via dynamic knowledge retrieval. To mitigate catastrophic forgetting, we construct a large-scale offline multimodal database—enabling continual adaptation through retrieval of similar cases during training. We conduct a comprehensive validation of our proposed framework against existing state-of-the-art methods; extensive experiments across 11 medical datasets, covering 9 modalities and 10 organs, demonstrate that our approach achieves state-of-the-art accuracy.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0321_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{QinChe_Diagnose_MICCAI2026,
        author = { Qin, Chendong AND Huang, Shuai AND Yao, Yifei},
        title = { { Diagnose Like a Doctor: Misdiagnosis-Aware Structured Memory for Continual Learning of Medical Vision-Language Models } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes a misdiagnosis-aware continual learning framework for medical vision-language models, where a hierarchical “Pitfall Graph” encodes disease mimics and exclusion criteria to guide retrieval-augmented training. The method combines graph-guided retrieval from a large-scale multimodal database with visual-semantic reranking and logit distillation to mitigate forgetting and improve fine-grained discrimination. Experiments on 11 datasets across multiple modalities show consistent improvements over recent continual VLM baselines, with notable gains in accuracy, AUC, and reduced forgetting.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1:Clinically motivated formulation. The proposed Pitfall Graph explicitly models disease mimics and exclusion criteria, introducing “negative diagnostic knowledge” into the learning process. This is well aligned with real clinical reasoning, where errors often arise from confusion between visually similar conditions rather than missing knowledge. 2: Technically non-trivial integration. The framework combines graph-guided query generation, retrieval from a large-scale multimodal database, visual-semantic reranking, and dual-model distillation within a continual learning setup. This goes beyond standard replay or RAG pipelines and provides a coherent mechanism to shape decision boundaries during sequential updates. 3: Strong reported empirical performance. The method shows consistent improvements over recent continual VLM baselines across 11 datasets, with gains in ACC/AUC and less negative BWT, particularly under challenging random task order settings.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1: The paper’s mechanistic conclusion is not fully isolated by the experiments. The motivation for the method is that modeling mimics and exclusion criteria may improve fine-grained discrimination beyond conventional positive-only retrieval. This is an interesting idea, and the proposed Pitfall Graph is a meaningful way to encode such information. However, the experiments do not include a direct comparison to a standard RAG/retrieval-only variant using the same external database and training setup. The closest ablation (w/o HPG) removes the graph component but still keeps retrieval, reranking, and distillation, so it does not cleanly isolate how much benefit specifically comes from the misdiagnosis-aware structure. Therefore, while the overall method performs well, the paper does not yet provide sufficiently controlled evidence for the stronger conclusion that the gains are specifically driven by modeling mimics/exclusion criteria rather than by the broader retrieval-and-distillation framework. 2: Potential confounding from large external memory. The method relies on a large-scale external multimodal database (Open-PMC-18M). Competing baselines do not appear to have access to an equivalent memory source, making it difficult to disentangle whether improvements stem from the proposed formulation or simply from additional data and retrieval capacity. 3: Evaluation setting is weaker than the clinical claims suggest. The experiments are conducted under a task-incremental learning setting with known task boundaries, which is substantially easier than class-incremental or task-agnostic scenarios. Despite the “diagnose like a doctor” framing, evaluation remains limited to benchmark classification without clinical validation, reader studies, or deployment-oriented analysis. 4: Reproducibility is limited. The manuscript does not provide sufficient implementation details to reproduce the method reliably, and no code repository is provided at the time of submission.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses an important problem and proposes an interesting idea: incorporating mimics and exclusion criteria into a structured retrieval framework for continual medical VLM learning. The reported empirical results are strong, with broad evaluation across multiple datasets, modalities, and task orders. However, the paper does not yet provide sufficiently controlled evidence to isolate the contribution of the proposed misdiagnosis-aware structure from the broader retrieval, reranking, and distillation pipeline. In addition, the novelty relative to related retrieval-guided and graph-enhanced medical learning approaches is not fully clarified, and the current manuscript does not provide enough detail for reproducibility. Overall, I find the work promising and technically relevant, but currently slightly below the acceptance threshold without a stronger experimental and methodological justification.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I thank the authors for the rebuttal. The clarification that the “w/o HPG” ablation preserves the same external database, retrieval, reranking, and distillation components while removing only the HPG structure partially addresses my main concern.

    Some concerns still remain. A positive-only retrieval baseline with the same memory would provide a cleaner comparison. In addition, the evaluation is still limited to task-incremental learning, and reproducibility remains limited without released code.

    Overall, the rebuttal sufficiently alleviates my primary concern, and I support acceptance.



Review #2

  • Please describe the contribution of the paper

    The authors propose a “Misdiagnosis-Aware Structured Memory” framework designed to tackle catastrophic forgetting and fine-grained feature confusion in the Continual Learning (CL) of medical Vision-Language Models (VLMs). The main contributions are: 1.The introduction of a Hierarchical Pitfall Graph (HPG) that links medical entities not just to their semantic definitions, but explicitly to their visual mimics and exclusionary criteria. 2.A Retrieval-Guided Continual Training framework that queries a massive 18-million sample offline database (Open-PMC-18M) during training. This acts as a dynamic memory bank to stabilize model updates through dual-model logit distillation, rather than relying on a static rehearsal buffer. 3.Comprehensive state-of-the-art validation across 11 diverse medical datasets covering 9 modalities and 10 organs.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Highly Novel Formulation: The construction of the Hierarchical Pitfall Graph is an excellent, intuitive contribution. Standard retrieval models focus on reinforcing positive semantic matches, overlooking hard negatives. Explicitly encoding diagnostic logic to teach the model what a disease is not directly mirrors human clinical cognitive workflows.

    Extensive and Robust Evaluation: The experimental design is exceptionally thorough, testing the method across 11 datasets spanning 9 imaging modalities (including CT, MRI, Ultrasound, Histopathology, etc.).

    Strong Empirical Results: The proposed method achieves state-of-the-art accuracy, significantly outperforming recent baselines like GIFT, ZSCL, and CoOp in both modality-wise and random order setups. For example, in the challenging random order setting, it achieves 71.68% Accuracy and 88.79% AUC.

    Scalable Rehearsal Strategy: Bypassing traditional, limited-capacity static rehearsal buffers by dynamically querying a large-scale multimodal database (Open-PMC-18M) represents a strong step forward for large-scale medical CL.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Computational Overhead: The framework relies on continuously querying an 18-million sample database and performing Visual-Semantic Reranking during training. The paper lacks a detailed discussion on the computational complexity, training time, and memory overhead introduced by this dense retrieval step.

    Graph Construction Noise: The HPG relies on unstructured clinical evidence mined from PubMed. The paper does not thoroughly discuss the potential noise, hallucinations, or error-propagation risks involved in mining these exclusionary edges at scale.

    Reliance on Proprietary Models for Main Experiments: While the authors thoughtfully provide an ablation study using open-source models (like Qwen-3-32B and DeepSeek-V3) , the primary experimental backbone relies on the proprietary GPT-4 model. This can restrict immediate reproducibility.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    This is a highly impressive and well-motivated paper. Bridging the gap between standard continual learning metrics and actual clinical reasoning, specifically through the lens of ruling out misdiagnoses/mimics, is a significant contribution to the MICCAI community.

    To further strengthen the manuscript, consider adding a brief subsection detailing the computational throughput (e.g., iterations per second) of your method compared to standard continual fine-tuning, as querying Open-PMC-18M in the loop is likely an expensive operation.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I strongly recommend accepting this paper. It directly tackles the critical “stability-plasticity” dilemma in medical VLM Continual Learning. The introduction of the Hierarchical Pitfall Graph is structurally innovative, allowing the model to leverage adversarial connections (mimicry and exclusionary edges) to build sharper decision boundaries. The evaluation is exhaustive, covering 11 different datasets and demonstrating clear superiority over existing state-of-the-art CL methods. The minor weaknesses regarding computational cost documentation and graph-mining noise do not outweigh the methodological ingenuity and strong empirical outcomes of the work.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper proposes a misdiagnosis-aware structured memory framework for continual learning of medical VLMs. It builds a large-scale offline multimodal database for similar-case retrieval during training. It further introduces a hierarchical Pitfall Graph that links medical entities to their definitions, common mimics, and exclusion criteria.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper offers an interesting perspective by framing database case retrieval around the way experienced clinicians make diagnoses, namely through careful exclusion of potential mimics. 2.The experimental evaluation includes comparisons against recent continual-learning baselines for medical VLMs.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Method motivation and experimental fairness. I am not fully convinced by the use of an external medical knowledge database in the VLM-CL setting. Conceptually, this seems closer to a retrieval- or replay-based strategy, since it assumes that similar and known exemplars are available in the database. In a more clinically realistic continual-learning scenario, both models and human experts would need to learn unseen knowledge rather than rely on retrieval from a predefined database. This also raises a fairness concern in the experiments, since baselines such as LwF, iCaRL, and MoE-CL do not appear to have access to a comparable external knowledge source or replay mechanism, which may partly explain the performance gap. 2.Missing dataset-task information. In Section 3.1, a clearer definition of the task associated with each dataset would make the experimental protocol easier to understand. 3.Unclear terminology around knowledge replay. Some terms related to the retrieval mechanism are not clearly defined. In Dense Retrieval and Visual-Semantic Reranking, the phrase “dynamically retrieve historical exemplars” is not entirely clear. The paper would benefit from a more explicit distinction between this knowledge replay stream and standard rehearsal replay in continual learning. In particular, it should clarify that no old-task samples are explicitly stored and replayed, and that the method instead relies solely on retrieval from an external multimodal database. Similar ambiguity also appears in the use of “historical exemplars” in the Introduction and Fig. 2, and “historical concepts” in the Dual-Model Logit Distillation section.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    My score is mainly based on the concerns outlined in the weaknesses above, but I would be open to revising it if these concerns are adequately addressed.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    I am still not convinced of the necessity of using external memory/retrieval, nor of the fairness of the comparison with other baselines when the proposed method is assisted by such external memory.

    I am also not convinced by the task-incremental classification setting where the task identity is available at inference. In a realistic continual-learning clinical scenario, the model is expected to perform well across all previously learned tasks without being given explicit information or hints about the task ID.



Author Feedback

We thank the reviewers and meta-reviewer for their constructive comments. We address the main concerns: HPG contribution/novelty, retrieval fairness, evaluation scope/task definition, overhead/noise, and reproducibility. 1) HPG contribution and novelty (R1/R3). We agree that isolating the source of improvement is important. The submitted Table 2 ablation provides relevant evidence: “w/o HPG” keeps the same external database, retrieval, visual-semantic reranking, and dual-model distillation, while removing only the HPG-guided mimic/exclusion structure. Its consistent degradation under both task orders indicates that HPG contributes beyond generic retrieval/distillation. A positive-only RAG baseline is useful future work, but the submitted ablation already tests whether the misdiagnosis-aware structure adds value beyond the remaining pipeline. The novelty is not simply RAG/replay, but encoding negative diagnostic knowledge, i.e., mimics and exclusion criteria, into retrieval topology, so retrieved knowledge emphasizes discriminative boundaries rather than only positive disease definitions. 2) External memory, fairness, and replay terminology (R1/R3). We compare with representative CL and continual VLM baselines under the same task-incremental protocol. Our goal is retrieval-augmented CL, not memory-free CL. Open-PMC-18M is used as an offline non-parametric biomedical image-text memory, not as supervised storage of old target-task samples or target-dataset labels. No previous target-task samples are saved or replayed. Thus, our “knowledge replay stream” is not standard rehearsal replay: “historical exemplars” are image-text pairs retrieved from Open-PMC-18M, and “historical concepts” are teacher responses on these retrieved exemplars for distillation. These samples serve as semantic/visual anchors, not additional labeled training data. Since external retrieval is part of our framework, the key fairness evidence is the retrieval-preserving HPG ablation: with the same memory and pipeline, removing the graph still weakens performance, indicating gains beyond external data. We clarify this terminology here and will use it consistently if accepted. 3) Evaluation scope, task definition, and clinical claims (R1/R3). The paper follows the task-incremental formulation in Sec. 2.1, where tasks arrive sequentially with task-specific label spaces. In Sec. 3.1, the task unit is the dataset rather than an individual class: each of the 11 datasets defines one sequential diagnostic classification task with its own label space, covering multiple modalities and organs. We agree that class-incremental/task-agnostic CL is more challenging, and benchmark classification does not establish clinical deployment readiness. “Diagnose like a doctor” is a design motivation: using mimics and exclusion criteria to reduce fine-grained confusion. We do not claim clinical validation, reader-study performance, or deployment readiness. 4) Overhead and graph noise (R2). Retrieval does not require repeatedly training on all 18M samples. The database is indexed offline; dense retrieval selects a small candidate set, and VSR reranks only these retrieved candidates. Thus, the added cost mainly comes from top-k retrieval/reranking, not full-database optimization. For graph noise, HPG combines UMLS/MeSH with PubMed evidence. Noisy edges affect retrieval prompts rather than labels, and VSR further reduces their impact by filtering visually inconsistent candidates. The w/o HPG ablation and Fig. 5 show that HPG remains useful despite possible noise. 5) Reproducibility and proprietary models (R1/R2). We acknowledge that no code repository was provided at submission. Still, Sec. 3.1 reports the backbone, LLM, hardware, hyperparameters, iterations, and dataset/task sources. Table 2 includes open-source LLM/backbone variants, mitigating concerns about dependence on GPT-4.If accepted, we will release code and settings.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    For the rebuttal, the authors should analyze the contribution of the proposed misdiagnosis-aware structure, clarify the novelty, and provide enough details for reproducibility and fairness of experiments.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper addresses continual learning for medical VLMs by introducing a misdiagnosis-aware memory structure that models disease mimics and exclusion criteria. Although concerns remain about reliance on external retrieval memory and task-incremental evaluation, the rebuttal clarified that the HPG ablation preserves the same retrieval pipeline while removing only the misdiagnosis-aware structure, supporting its added value. Overall, the work is technically relevant and sufficiently justified for acceptance.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper addresses an important continual medical VLM learning problem and proposes a clinically meaningful misdiagnosis-aware structured memory via mimics and exclusion criteria. Although concerns remain about external retrieval memory and the task-incremental setting, the rebuttal reasonably clarifies that the HPG ablation preserves the same memory/retrieval pipeline and isolates the contribution of the misdiagnosis-aware structure. Overall, the empirical breadth and post-rebuttal reviewer balance support acceptance.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    After rebuttal, the overall reviewer opinions are mixed but lean positive. Reviewer #1 supports acceptance after the authors clarified that the “w/o HPG” ablation keeps the same external database, retrieval, reranking, and distillation components while removing only the HPG structure, partially isolating the contribution of the proposed misdiagnosis-aware graph. Reviewer #2 strongly supports acceptance, emphasizing the novelty of modeling mimics and exclusion criteria and the broad evaluation across 11 medical datasets. Reviewer #3 remains negative, mainly due to concerns about the necessity and fairness of external memory/retrieval and the realism of the task-incremental setting with task identity available at inference.

    Overall, the paper remains borderline, but the rebuttal sufficiently addresses the central contribution concern for a majority of reviewers. I therefore recommend acceptance, with the expectation that the final version clearly discusses the limitations regarding external-memory fairness, task-incremental evaluation, computational cost, and reproducibility/code release.



back to top