Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

This work proposes a novel framework for analyzing disease progression using time-aware neural ordinary differential equations (NODE). We introduce a “time-aware head” in a framework trained through self-supervised learning (SSL) to leverage temporal information in latent space for data augmentation. This approach effectively integrates NODEs with SSL, offering significant performance improvements compared to traditional methods that lack explicit temporal integration. We demonstrate the effectiveness of our strategy for diabetic retinopathy progression prediction using the OPHDIAT database. Compared to the baseline, all NODE architectures achieve statistically significant improvements in area under the ROC curve (AUC) and Kappa metrics, highlighting the efficacy of pre-training with SSL-inspired approaches. Additionally, our framework promotes stable training for NODEs, a commonly encountered challenge in time-aware modeling.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0263_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0263_supp.pdf

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Zeg_LaTiM_MICCAI2024,
        author = { Zeghlache, Rachid and Conze, Pierre-Henri and El Habib Daho, Mostafa and Li, Yihao and Le Boité, Hugo and Tadayoni, Ramin and Massin, Pascale and Cochener, Béatrice and Rezaei, Alireza and Brahim, Ikram and Quellec, Gwenolé and Lamard, Mathieu},
        title = { { LaTiM: Longitudinal representation learning in continuous-time models to predict disease progression } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15005},
        month = {October},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The paper introduces a novel framework that integrates time-aware neural ordinary differential equations (NODEs) with self-supervised learning (SSL) for analyzing disease progression, specifically focusing on diabetic retinopathy. It employs a time-aware component within the NODE architecture to better utilize temporal data, aiming to improve prediction accuracy.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper’s main strength lies in its innovative integration of Neural Ordinary Differential Equations (NODEs) with Longitudinal Self-Supervised Learning (LSSL) to enhance the modeling of disease progression. This approach is novel in its application of NODEs within an SSL framework, particularly for capturing dynamic changes over time in diabetic retinopathy. The use of NODEs allows for the efficient handling of irregular time series data, which is common in clinical datasets, thereby improving the model’s applicability to real-world medical scenarios. Additionally, the demonstration of clinical feasibility through statistically significant improvements in AUC and Kappa metrics substantiates the model’s potential utility in enhancing disease prediction accuracy.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The AUC results for diabetic retinopathy over 1, 2, and 3 years are significantly lower than existing literature, raising concerns about the model’s clinical utility. The consistency of results across these intervals contradicts typical expectations from progressive diseases, suggesting potential shortcomings in the model’s ability to effectively capture temporal dynamics. Although the integration of Longitudinal Self-Supervised Learning (LSSL) with Neural Ordinary Differential Equations (NODEs) has been previously explored, this paper does not sufficiently differentiate its approach or demonstrate clear improvements over these existing methodologies. The lack of publicly available source code complicates the verification and reproduction of the study’s results, reducing transparency and limiting the ability of the research community to independently validate and build upon the findings. Additionally, the absence of a second, independent dataset for external validation limits the assessment of the model’s generalizability across different clinical settings and patient populations. This is crucial for confirming the model’s efficacy beyond the initial experimental setup. Finally, the study focuses on a specific NODE implementation but does not compare its performance against other potential architectures. Exploring different architectures could provide deeper insights into the model’s efficiency and effectiveness, offering a broader perspective on its capabilities and limitations in various scenarios.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    No

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. The reported AUC results are lower than those typically found in the literature. It is recommended that the authors investigate the underlying reasons for these suboptimal outcomes. Potential areas to explore include the model’s parameter tuning, the inclusion of more diverse features, or adjustments to the NODE architecture to better capture the progressive nature of the disease. 2.The integration of LSSL with NODEs has been previously explored. The authors need to better highlight what distinguishes their approach from existing methods. Specific enhancements or unique applications should be detailed to emphasize the novelty of the work. 3. To enhance the paper’s credibility and allow for community verification, it is crucial to make the source code available. Open sourcing the code will aid in reproducing the results and encourage further research and modifications by the community. 4. The absence of external validation limits the generalizability of the findings. The authors should consider using an independent dataset to validate their model, which would strengthen the case for its clinical applicability and robustness. 5. The study could be enriched by comparing the proposed NODE-based model with other architectures or backbones. Such comparisons might provide insights into the effectiveness of different approaches and help justify the choice of NODEs for this application.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please see the ‘weakness’ session and comments session.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper introduces an innovative framework specifically designed to enhance the pre-training of time-aware models for monitoring disease progression, with a focus on diabetic retinopathy.

    The core contribution is the integration of NODEs within SSL, (SimCLR, BYOL), to model disease progression. This combined approach, mirrors the temporal and progressive nature of clinical assessments more accurately than traditional predictive models that ignore time-based changes. The method is generic across various medical and healthcare domains.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    1. Novelty and Relevance: The integration of NODEs with SSL is innovative and directly applicable to clinical settings, potentially improving outcomes in managing diabetic retinopathy.

    2.Methodology: The authors comprehensively detail the mathematical framework that effectively integrates Neural Ordinary Differential Equations (NODEs) with self-supervised learning (SSL). The incorporation of time-aware components into modeling represents a crucial advancement and has significant potential for broader application.

    1. Experimental Design: Utilizing a substantial dataset from the OPHDIAT database, which includes data from about 100,000 patients collected over more than a decade, provides a robust test bed for the model.

    2. Practical Implementation: The model operates efficiently on a single NVIDIA A6000 GPU, showcasing its feasibility for practical clinical deployment without extensive computational resources.

    3. Results: The model demonstrates significant improvements in predictive accuracy over baseline models, evidenced by superior AUC and Kappa metrics, underscoring the effectiveness of the proposed method in a real-world healthcare application.

    Overall, this work offers a significant contribution to predictive healthcare analytics, particularly for conditions that require monitoring over extended periods. This paper serves as an exemplary case of how integration of time-awareness component can be effectively applied to clinical diagnostics, suggesting its utility and adaptability for other medical scenarios as well.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The paper offers a comprehensive evaluation, but a closer look at the results in Table 1 reveals that BYOL consistently outperforms SimCLR, except in the third year’s AUC 3 prediction. Given that predictions for Year 3 might be inherently more challenging, it would be insightful if the authors could discuss whether the inclusion of negative samples in SimCLR might contribute to a more robust model for longer-term predictions compared to the BYOL approach, which does not use negative samples.

    While NODEs demonstrate improved outcomes in many scenarios, there is a potential dependency on the solver used, specifically the “dopri5” or Dormand-Prince Method. This reliance could pose risks regarding the solver’s specificity, especially in the face of biological time variability which might introduce discontinuities and singularities in data representation, particularly concerning the time component. Addressing these issues requires a detailed examination of how different solver characteristics might impact the model’s performance. It would be beneficial for the authors to explore this aspect more thoroughly, potentially by comparing performance across different solvers or discussing strategies to mitigate solver-specific biases in the context of the biological variability inherent in the data.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    No.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Overall, this paper is innovative, particularly in its integration of time-aware components into the modeling process, which is highly relevant and aligned with clinical approaches to understanding disease progression.

    In the benchmark comparisons presented in Table 1, I noticed an unexpected result where RESNET50 outperformed RESNET50+NODE (rows 1 and 2). Typically, one might expect the inclusion of NODE to enhance the model’s performance due to its ability to handle time-dependent data. It would be beneficial if the authors could provide an explanation for why this was not the case.

    Figures 1 and 2: There is room for improvement in Figures 1 and 2 to enhance clarity for readers. Specifically, Figure 1 would benefit from greater visual contrast between the pairs (a, c) versus (b, d) and between figure (e) versus (f), helping to distinguish the different conditions or methods being compared. Figure 2, on the other hand, appears cluttered and could be confusing. Simplifying the presentation or providing a clearer separation of elements could help in making the information more accessible and understandable. Also, typo in Figure 2, for legends: Similary criteria -> Similarity criteria.

    Additionally, the paper lacks information regarding data or image pre-processing. While I assume that basic or standard processing techniques were applied, it would be valuable for the paper to include details about any pre-processing steps that were undertaken. This information would aid in the reproducibility of the study and provide clarity on the initial conditions under which the models were trained.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a clinically significant problem with technically solid mathematical formulation. The incorporation of a time-aware component using Neural Ordinary Differential Equations (NODEs) is particularly pertinent for clinical diagnosis. The evaluation appears thorough; however, some additional clarifications regarding the data used and the presentation of results would enhance the overall clarity and impact of the findings.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper leverages time-aware neural ordinary differential equations to develop a novel framework for analyzing disease progression. Evaluations are performed on fundus photographs.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1) Novel application of neural ordinary differential equations. 2) Use of large dataset for evaluation. 3) Clear methods descriptions. 4) Use of publicly available data, which in combination with clarity/details in methods description, enhances the paper’s reproducibility. 5) Extensive performance evaluations.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1) Without standard deviations or confidence intervals, and p-values for comparisons, the statistical significance in the comparisons shown in Tables 1-3 is unclear. 2) The reported performance is moderate.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Strong submission despite a fee weaknesses.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Methodological novelty, large-scale evaluation.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

First, I would like to thank the reviewers for their time and the feedback they provided for our paper. Their insights will be crucial for the extended version of this work. For the rest of the rebuttal response, we will refer to reviewers in the order of appearance, such as #R1, #R2, and #R3. We are also pleased to hear that all three reviewers think our paper presents a promising idea with great potential, especially #R1 and #R2.

Enhancement

  • While we are not permitted to share the code, you can directly ask any questions. I will personally answer any questions related to the implementation or details related to the training of the proposed framework. You can contact me at the following address: rachid.zeghlache@univ-brest.fr.
  • As indicated in the paper and pointed out by reviewer #R3, the use of LSSL and NODE was explored [1]. However, to the best of our knowledge at the time, there were no papers that combined common SSL paradigms (such as BYOL and SimCLR) with NODE for pre-training purposes. We will make sure to emphasize this point.
  • We agree with #R1 that there is certainly room for improvement in the figures. We will make sure to take into consideration the advice received to increase understanding and clarity.

Correction

  • We will make sure to correct the typo for the camera-ready paper as pointed out by #R2 inside the main figure.
  • We will also add the confidence intervals for the different p-values given in the paper to provide a better overview of the model’s performance compared to the baseline, as suggested by #R2.
  • We agree with #R1 including the detail related to the preprocessing would help. We indeed use classic image processing techniques such as cropping, normalization, and classic data augmentation. These details will be added to the manuscript.

Extension

  • The reviewers have pointed out many ways to enhance the value of the paper. We will make sure to add these elements in the journal extension of the paper. We thank the reviewers for their suggestions, such as the comparison of other methods and external datasets suggested by #R3. Additionally, we will compare different types of solvers to understand their importance and impact on the proposed method. We will also conduct a more in-depth analysis of the impact of the number of negatives for the SimCLR-based framework. Finally, a rigorous hyperparameter search will be conducted in the extension to demonstrate the importance of each parameter and to try to improve our current results as suggested by #R3.
  • We also think that using an external dataset to confirm the robustness and generalizability of our method will add greater weight to the proposed framework and convince his utility. For this matter, we are open to collaboration if someone wishes to test our method on their dataset.
  • We appreciate #R2 highlighting the unexpected performance in Table 1 (RESNET50 vs. RESNET50+NODE). We’re investigating potential causes such as data preprocessing, hyperparameter tuning, and dataset complexity. We will conduct further experiments (preprocessing for NODE compatibility, hyperparameter search, and temporal characteristic analysis) to clarify this and improve RESNET50+NODE performance. The manuscript extension will include these investigations.

[1] Zeghlache, Rachid, et al. “Longitudinal self-supervised learning using neural ordinary differential equation.” International Workshop on PRedictive Intelligence In MEdicine. Cham: Springer Nature Switzerland, 2023.



Meta-Review

Meta-review not available, early accepted paper.



back to top