Abstract

Longitudinal analysis of a patient’s screening history is fundamental to mammography interpretation. Yet existing deep learning models struggle with the irregular, continuous-time nature of screening data, where patient histories involve unevenly timed multi-view exams paired with evolving textual reports. To address this gap, we introduce \textbf{Dynamo}, a novel multi-modal pretraining \underline{\textbf{dyna}}mic framework for longitudinal \underline{\textbf{m}}amm\underline{\textbf{o}}graphy with two key innovations to model exam timelines as latent trajectories, i.e., a coarse-fine grained exam-level temporal encoder based on Neural Controlled Differential Equations (Neural CDEs) and a Temporal Visual Question Answering (TVQA) loss for query-driven masked token prediction conditioned on temporal context. We perform comprehensive evaluations on downstream tasks including risk prediction using large-scale datasets (EMBED and CSAW-CC), BI-RADS assessment, and breast density classification. Across all benchmarks, Dynamo achieves overall gains over state-of-the-art vision-only and CLIP-style models, improving both calibration and temporal reasoning. On the EMBED dataset, representative gains include $11.81\%$ relative reduction in Risk Prediction’s Brier Score, $3.59\%$ relative improvement in BI-RADS $\kappa$, and $2.39\%$ relative improvement in BI-RADS AUC in zero-shot settings. Our code is available at \href{https://github.com/Mortal-Suen/Dynamo}{this URL}.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1911_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/Mortal-Suen/Dynamo

Link to the Dataset(s)

EMBED dataset: https://aws.amazon.com/marketplace/pp/prodview-unw4li5rkivs2#overview%7D CSAW-CC dataset: https://researchdata.se/en/catalogue/dataset/2021-204-1

BibTex

@InProceedings{SunZij_From_MICCAI2026,
        author = { Sun, Zijun AND Ge, Enze AND Thrun, Solveig AND Kampffmeyer, Michael},
        title = { { From Exams to Trajectories: Neural Controlled Differential Equations for Longitudinal Mammography Vision-Language Pretraining } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16887},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a multi-modal approach to mammography using a “continuous-time longitudinal encoder” and a temporal VQA as main novelties.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The problem solves a difficult problem and reports improvements to an important application.
    • The idea of using continuous-time encoder is new in this setting.
    • There is an ablation study on the different components (Table 3)
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The paper is impossible to read when printed. I suspect it violates guidelines by making font sizes too small in tables. As a further example, Fig 2(a) is very hard to see even in the pdf when zoomed-up and the different models are hard to compare.

    • The intuition behind the continuous-time advantage is not well described/motivated. The source data is inherently at discrete time events so the natural choice is to model it as such. Right now this is disregarded due to “heuristic”. The new approach is in my opinion also heuristic. Then, on the other hand, there is nothing wrong with heuristics! However, one would like to understand why it works better and what priors it brings.

    • Given that there is not any new technical novelty, the evaluation becomes more important. First, it is not clear whether the numbers come from previous work or if it is re-implemented. Then, I have tried to compare to previous work, for instance, Mirai [32], which the authors claim to follow, but the experiments in the present paper do not seem to be conducted in the same manner even though it is using the same data. Training and test sets are not described.

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper deals with an important problem and shows promising empirical results. However, the current presentation raises concerns regarding clarity and evaluation. In particular, key experimental details are missing, making it difficult to assess the fairness of comparisons and the relation to prior work (e.g. Mirai).

    Additionally, the methodological contribution is somewhat incremental, and the motivation for the continuous-time formulation is not sufficiently convincing.

    Overall, while the work has potential, the current version does not yet meet the bar for acceptance.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I am fine with accepting this paper. The authors wrote a good rebuttal, particularly regarding the motivation of a continuous model.



Review #2

  • Please describe the contribution of the paper

    This paper presents a novel framework for longitudinal breast cancer prediction using mammograms and associated text reports. The proposed framework extracts image embeddings through a pretrained image encoder (Mirai) and generates temporal-informed embeddings for each patient using a neural controlled differential equations block. Masked text reports are encoded using Bio-Clinical BERT and used for temporal visual question answering (TVQA). Predictions are made using a gated fusion approach of both modalities. For training, besides the standard cross-entropy loss, the authors propose adding auxiliary contrastive loss functions to reinforce patient-specific embeddings, breast-view asymmetry, and image-text alignment. The framework is tested with one main dataset and compared with other works and methodologies, in both risk ranking, calibration and time dependent AUCs. This work improves upon current approaches and provides useful insights into best practices and design choices for longitudinal breast cancer prediction.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper is well written and structured
    • The proposed multimodal framework effectively combines mammographic images with masked text reports for risk prediction
    • The design of the auxiliary losses has clinical relevance (asymmetry aware loss)
    • The experimental evaluation is comprehensive, including comparisons with multiple baselines across several metrics (e.g., WMAE, Brier score, c index, and time dependent AUCs)
    • The ablation study is comprehensive and offers insights into the contribution of each component
    • The reported results are promising and competitive compared with existing methods
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • There is no formal statistical comparison with competing methods (e.g., confidence intervals or significance tests), which limits the robustness of the performance claims
    • The paper provides limited information about the dataset distribution and follow-up times, which makes it harder to assess generalizability
    • Only one dataset is used to report metrics. The results obtained with another dataset are summarized in a figure with no numerical information
    • Computational aspects are not reported, such as training and inference times, or the number of trainable parameters, which are important for real world deployment considerations
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • Figure 1 contains typos (e.g., “Maskd”, “Hideen”) that should be corrected for clarity
    • In Table 1 the reported Brier scores appear to be in percent, but it’s not indicated
    • Section 3.3 does not mention Table 2 (second paragraph)
    • The dataset splitting and any temporal or patient wise partitioning are not described
    • Figure 2a is visually dense and could benefit from simplification to improve readability or replacement with another form of reporting
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a strong, well motivated, and experimentally validated multimodal framework for longitudinal breast cancer risk prediction. The combination of image embeddings, temporal modeling, and text based TVQA is innovative and clinically relevant. The ablation and comparison studies are convincing, and the overall presentation is clear. The main weaknesses are mostly related to missing details, which could be addressed in a minor revision, without affecting the main contribution.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    interesting methodology and rather complete evaluation



Review #3

  • Please describe the contribution of the paper

    This paper introduces Dynamo, a multimodal pretraining framework for longitudinal mammography that models patient screening histories as continuous-time latent trajectories using Neural Controlled Differential Equations . The approach combines a coarse-to-fine Neural CDE temporal encoder to handle irregular exam intervals and a Temporal Visual Question Answering (TVQA) objective to align imaging with report text over time. The method is evaluated on large-scale datasets across multiple downstream tasks including risk prediction, showing improvements over vision-only and vision-language modeling.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper clearly identifies two important gaps: discrete-time modeling limitations for irregular screening intervals and lack of temporal grounding in vision-language mammography models. 2.The TVQA objective is an interesting contribution: It allows temporal reasoning in language and provides a more structured supervision signal. 3.There is a complete experimental validation that includes multiple datasets, diverse tasks, ablations and robustness analysis to irregular sampling. 4.There is a use of clinically relevant evaluation metrics like the use of Risk Prediction’s Brier Score, C-index, and long-horizon AUC.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The EMBED dataset only uses synthetic reports. This may limit the realism of language supervision like having a potential bias toward structured patterns. This limitation is acknowledged but not deeply analyzed. 2.There is little discussion of efficiency, training cost or inference speed, which is important for clinical adoption. 3.The robustness analysis is a strong point, but could include more realistic missing-data scenarios. How does Dynamo perform with real radiology reports instead of pseudo-reports? 4.Can the authors provide qualitative examples demonstrating temporal reasoning (e.g., report changes over time)? How sensitive is the model to irregular sampling density in extremely sparse histories?

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper presents a well-motivated and technically solid approach to longitudinal multimodal modeling in mammography. There is a meaningful clinical relevance.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    I am happy with the explanation on sensitivity to irregular sampling computational. The proposed changes talking about the challenges of not finding real datasets, to revise figure 1 and to mention the computational aspects will improve the final paper.



Author Feedback

We thank the reviewers for the feedback and will add clarifications. We are glad they appreciate the difficulty and importance of the problem(R1,R3), the novelty of leveraging a continuous-time encoder(R1), our clinical relevant losses(R2), and comprehensive empirical studies(R1,R2,R3).

*Intuition behind continuous-time advantage (R1) While exams are discrete, the underlying disease evolution is continuous and sampled at irregular intervals. Our approach explicitly captures this by integrating a latent state over real elapsed time, imposing a smooth-in-time prior that improves robustness to irregular sampling and missing visits, versus discrete models with ad-hoc positional encoding. Empirically, replacing NCDE with a Transformer degrades i.e. C-index from 82.79 to 75.38 (Table 3), and NCDE is markedly more stable under visit-dropout/irregular regimes (Fig. 2b). We will improve this description.

*Novelty (R1) Clinically, disease progresses continuously while exams are irregular, and interpretation combines stable phenotypes with subtle, view-specific asymmetries. We encode these priors by introducing the first exam-level Neural CDE for longitudinal mammography, using coarse (phenotype) and fine (asymmetry/view-local) streams, augmented with trajectory‑conditioned TVQA and explicit asymmetry regularization.

*Evaluation and dataset distribution (R1,R2) To ensure a fair comparison, we adopt a common protocol instead of reusing prior numbers. All baselines are trained and/or evaluated on the same EMBED/CSAW-CC patient wise 70/10/20 train/val/test splits as Dynamo. Multiple follow up visits are available for approx. 64% (EMBED) and 75% (CSAW-CC) of patients. Differences from published Mirai style results can arise from cohort construction, horizons, and preprocessing. Mirai and AsymmMirai were directly evaluated using public final checkpoints. CLIP, DINOv2, Random-ViT, ALBEF, MedGemma, and Mammo-CLIP used public pretrained encoders, with downstream heads/finetuning on patient wise splits. OncoNet, VMRA-MaR, and MaMA were retrained as public final checkpoints matching the task were unavailable.

*Computational aspects (R2,R3) The final model has 121M parameters. Training used PyTorch FP16 DDP on 8 NVIDIA H200 GPUs for 30 epochs, batch size 256, requiring 132 GPU+CPU hours. Inference on 115,810 test samples required 0.3 h, or 0.0024 h per 1000 instances. We will add the computational aspects in the revision.

*Synthetic reports (R3) We follow prior work in synthesizing reports, due to the lack of available datasets. However, we agree that we could further elaborate the potential challenges with this approach and will do so in the future work section.

*Qualitative examples of report changes (R3) Fig.1 contains a concrete TVQA instance: the model is asked to recover a masked dynamic term describing a BI-RADS category change at a 1 year follow up, with candidate terms encoding opposite temporal directions. The prediction is conditioned on the NCDE state z_i, which summarizes the patient history up to exam i. We will revise Fig. 1 to highlight this example more clearly.

*Sensitivity to irregular sampling (R3) While we can see a noticeable drop when comparing the performance for the patients with the most regular screening intervals to the most irregular ones (Fig. 2b), this drop is considerably smaller for the proposed method compared to the common transformer-based approach. NCDE natively treats missing exams as longer integration intervals, which stabilizes performance in extremely sparse settings.

*Formal statistical comparison (R2) This is a fair comment and formal statistical comparisons were not reported due to the considerable computation requirements of these models.

*Typos, presentation, visually dense (R1,R2) Thank you for pointing out the typos. The choice of summarizing the CSAW-CC results in a figure was motivated by space constraints. We understand that this was sub-optimal and will revise it in the final version.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    Two of the reviewers recommended accept, while one reviewer recommended reject. The positive reviewers found the problem to be important and relevant, and the novelty of the contributions sufficient in this domain. The concerns from the negative reviewer include issues with legibility and motivation for continuous time modeling. The reliance on pseudo-reports is another concern. I recommend this paper is invited for a rebuttal to resolve these concerns.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    N/A

  • Please justify your recommendation.

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The authors provided a strong rebuttal, the paper is well-written, has interesting methodological contributions, and offers a strong evaluation.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    This paper proposes Dynamo, a framework for longitudinal analysis of mammography images with modeling of patient records (screening history) as continuous-time latent trajectories using Neural Controlled Differential Equations. Before the rebuttal, the reviewers asked for more details on the motivation for the continuous-time formulation, implementation, and computational aspects, as well as sensitivity to irregular sampling. The authors addressed these points in the rebuttal, providing clear explanations of the benefits of continuous modeling, especially for irregular screening intervals. Given the completeness of the experimental validation, the high interest of the gaps addressed in the paper to the community, and the originality of the methodological contribution, I recommand acceptance of this paper.



back to top