Abstract

Accurate prostate cancer grading is crucial for optimal disease management decisions, but current multiparametric MRI (mpMRI) based models rely on vast amounts of expert-labeled data. We propose a cross-modal latent supervision framework, where we aim to distill rich histopathology information into the mpMRI-encoder by self-supervised joint pretraining on unlabeled but patient-matched mpMRI and digitized prostatectomy data. Our framework jointly optimizes a unimodal mpMRI SimCLR objective to learn general mpMRI representations and a multimodal contrastive objective to align these mpMRI embeddings with a prostatectomy embedding obtained through a histopathology foundation model (PRISM). Finetuning this mpMRI encoder with a subset of labeled data demonstrates the effectiveness of our cross-modal supervision framework, outperforming both strong supervised and self-supervised mpMRI baselines for ISUP grading with 5.9 and 4.6 percentage points in quadratic weighted kappa, respectively. Our framework is registration-free and tolerant to data asymmetries, delivering substantial grading improvements when only 3% of mpMRI data is accompanied by its corresponding prostatectomy embedding during multimodal pretraining.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5738_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/DIAGNijmegen/crossmodal-supervision

Link to the Dataset(s)

N/A

BibTex

@InProceedings{SchDaa_Latent_MICCAI2026,
        author = { Schouten, Daan AND van der Laak, Jeroen AND Somford, Diederik AND Küsters-Vandevelde, Heidi AND Khalili, Nadieh AND Bosma, Joeran S. AND Saha, Anindo AND Twilt, Jasper AND Coelho, Tiago AND de Rooij, Maarten AND Huisman, Henkjan AND Veltman, Jeroen AND Yakar, Derya AND Litjens, Geert},
        title = { { Latent Lessons from Histopathology: A Registration-Free Cross-Modal Supervision Framework for mpMRI-Based ISUP Grading } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16891},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes a method that does not need to register histopathology images to mpMRI but can still do prostate cancer grading. The key idea of this paper is to distill histopathology information into an mpMRI encoder in latent space, thereby avoiding the need for histopathology registration to mpMRI images, as the task is difficult. The method combines a unimodal 3D SimCLR objective for learning general mpMRI representations with a multimodal contrastive objective that aligns mpMRI embeddings with prostatectomy embeddings obtained from a histopathology foundation model (PRISM). The framework is evaluated on large-scale prostate MRI data for both 6-class ISUP grading and binary csPCa detection, and shows clear improvements over both supervised training from scratch and unimodal self-supervised pretraining.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper addresses an important and well-motivated problem. Using pathology to supervise MRI images is clinically The two-branch design is elegant, the unimodal self-supervised branch learns mpMRI representations from large amounts of unlabeled MRI, while the multimodal branch supervised from pathology only where matched data are available. This makes the approach suited to situations where only a small subset of MRI studies has histopathology labels. The matched mpMRI-histopathology data is only presented to the model during training

    The experimental scale is also a strength. The authors combine a private institutional cohort with the PI-CAI training data, resulting in 13,075 unique patients and 18,859 MRI studies, with an internal test set of 2,298 studies.

    The improvements in results section are meaningful. On the internal test set, the proposed multimodal SSL framework improves QWK for ISUP grading from 0.617 (supervised) and 0.630 (unimodal SSL) to 0.676, while reducing MAE from 0.611 to 0.541.For binary csPCa detection, the paper reports further gains over both baselines, indicating that the learned representation is useful beyond the grading task alone.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The paper cites prior work using histopathology for MRI supervision, including registration-based or mixed-supervision approaches, but does not provide a more direct head-to-head comparison.

    The cross-modal supervision signal is derived from a relatively small paired subset of the development data: only 290 of 11,075 development patients have matched prostatectomy slides. While the paper demonstrates a measurable improvement from pathology-based cross-modal supervision on the internal test set, it remains unclear how robust this gain is and how well it would generalize beyond this limited paired subset.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (1) Strong Reject — must be rejected due to major flaws

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The author obtained a large scale private dataset and combined it with the PI-CAI training set. The paper addresses an important problem, proposes a practical registration-free framework, and shows consistent gains over baselines.My main reservations are that the novelty is moderate, since the framework mainly combines standard components, and the pathology supervision comes from only a small paired subset of the development data.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    This authors proposed a framework for joint mpMRI and whole slide prostate image pretraining without the need for registration, for the purpose of enhancing the accuracy of predicting ISUP grading of prostate cancer from MRI.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper tackles a clinically relevant problem. mpMRI is widely used in prostate cancer assessment, but biopsy remains the gold standard, so incorporating histopathology information into MRI representation learning is well motivated. The study also appears to use a reasonably sized prostate mpMRI dataset.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    The main issue is that the ablation study does not sufficiently validate the claimed benefit of the proposed framework. In particular, it does not clearly show that the improvement is truly driven by the use of histopathology information.

    The supervised 3D ResNet18 baseline may not be strong enough for a convincing comparison. It is unclear whether this baseline uses the same prostate gland mask-based cropping as the proposed self-supervised method. If not, the comparison may be confounded by differences in preprocessing rather than representation learning.

    The paper also lacks important implementation details, such as model capacity and hyperparameter settings, making it difficult to determine whether the performance gain comes from the proposed latent histopathology supervision or from architectural and optimization differences. More thorough ablations would be helpful, including random initialization and alternative definitions of positive and negative pairs.

    Another concern is potential bias in the histopathology data. The pathology images are obtained from prostatectomy specimens, which are typically associated with more severe cases and may therefore introduce bias into the learned model. This should be discussed more carefully, especially since the unimodal self-supervised and training-from-scratch baselines may be less exposed to this source of bias.

    The evaluation is performed only on an internal dataset, so the generalizability of the method to external cohorts remains uncertain

    The paper should also report the class distribution of ISUP grades, as this is important for interpreting the numerical results, especially if the class distribution is imbalanced.

    Sensitivity and specificity alone are difficult to interpret without specifying the operating threshold or providing a more comprehensive evaluation (for example, sens at controlled spec value, or vice versa).

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Figure 1 should be improved to make the text more readable.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall, the experiment does not sufficiently show the effectiveness of the proposed framework, that the performance gain is actually coming from the latent histopathology supervision and the design of the self-supervised learning scheme.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    I appreciate the author’s response, however, most of my questions remain unresolved. The concern about histopathology bias is partially addressed. It could be plausible that clinically insignificant cases in the development set may contribute to model performance, however, this did not answer the questions whether cross-modal supervision introduces a distributional bias toward prostatectomy-like, more severe, or treatment-selected cases. Class distribution is acknowledged but not resolved. The authors say they plan to report this in the final manuscript, but the current numerical results remain difficult to interpret whether performance is driven by imbalance. The external validation concern and the sensitivity/specificity interpretation also appears not answered. These questions are essential in medical imaging and should not be overlooked. The lack of external validation limits confidence in the generalizability of the proposed method, and sensitivity/specificity values are difficult to interpret without a clearly specified operating threshold or complementary metrics. Overall, the rebuttal relies heavily on promises of future clarification and additions to the final manuscript rather than providing new evidence. While these planned revisions may improve the presentation, they do not resolve the core methodological concerns at the rebuttal stage. The response does not sufficiently establish the source of the performance gain or address key concerns regarding bias, external validation, class imbalance, and evaluation.



Review #3

  • Please describe the contribution of the paper

    This paper proposes a hybrid unimodal and multimodal self-supervised learning (SSL) framework that achieves robust performance even in data-scarce settings where only 3% of the dataset includes paired pathology. A key contribution of this work is its registration-free, latent-space multimodal supervision strategy, which aligns mpMRI with histopathology without the need for spatial co-registration. The authors  validate their approach on a large institutional and multicenter dataset, demonstrating performance gains in both ISUP grading and binary clinically significant prostate cancer (csPCa) detection.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The key strengths of the paper are: -Registration-free multimodal formulation: The approach aligns MRI and histopathology data without requiring spatial co-registration. -Applicability to clinical data availability: The framework is designed to utilize unlabeled and weakly paired data. -Comprehensive experimental design: The evaluation is conducted on a dataset of 18,000 mpMRI studies and assesses performance across multiple tasks, specifically ISUP grading and csPCa detection. The methodology is compared against established supervised and unimodal self-supervised learning (SSL) baselines. -Quantitative performance increases: The proposed method yields measurable metric changes, including a 5.9-point increase in Quadratic Weighted Kappa (QWK) over the supervised baseline and a 4.6-point increase over the unimodal SSL baseline, as well as an increased AUROC for csPCa detection. -Tolerance to data asymmetry: The architecture supports learning environments characterized by a scarcity of paired histopathology-mpMRI samples.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Key limitations: -Limited exploration of alternative cross-modal designs The framework uses PRISM as the histopathology encoder, but does not compare against: joint end-to-end learned multimodal encoders, other contrastive paradigms (e.g., multi-instance contrastive MIL), or CLIP-style vision-language training adapted for pathology. This raises the question: Is performance driven by PRISM or by the framework itself?

    -No explicit handling of domain shift between institutions The model is trained across large multi-center mpMRI data, yet pathology foundation models are trained on heterogeneous slide datasets. No analysis is provided on domain shifts between: pathology slides from different scanners/hospitals, mpMRI protocol variability across PI‑CAI images.

    -Contrastive alignment is global (patient-level), Slide embeddings are max-pooled, which may lose fine-grain cancer heterogeneity. Unlike voxel-level registration methods (though more complex), this global alignment may bias representations for mixed-grade prostates.

    -Insufficient ablation on proportion of paired data. The study demonstrates performance when only ~3% paired data exist, but do not: show performance scaling curves (e.g., 1%, 5%, 10%, 30%) evaluate sensitivity to imbalance in grade distribution among paired pathology.

    -Latent-space collapse risks not discussed. With strong multimodal contrastive loss, there is potential for the mpMRI embeddings to overfit to histopathology embedding directions, harming purely mpMRI-driven tasks. No embedding-space analysis is shown.

    -Limited clinical interpretability assessment. While the goal is to “distill histopathology signals,” the paper does not provide a qualitative or quantitative assessment of what histopathology features the mpMRI encoder actually learns.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper contributes to an important and promising direction: multimodal SSL using histopathology to improve MRI-based prostate cancer grading without requiring registration. The empirical gains are meaningful, the dataset scale is compelling, and the framework is well-designed and practical. Some missing analyses, particularly around multimodal ablations, domain shifts, and interpretability, limit the merits of the paper. However, the contribution is notable and I believe it should be accepted.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers for their constructive feedback. We first address two shared concerns and then respond to each reviewer individually.

The first shared comment concerned the fact that our introduction highlighted related registration-based multimodal approaches, but that a head-to-head comparison with said methods was missing from the results section. First, bypassing the need for registration is precisely the goal and strength of our framework. We would be willing to compare against a registration-based framework, but the cited registration-based multimodal approaches (ref 1, 2) have not released code or models, which prevents this direct comparison. That leaves manual registration, which introduces observer bias and is infeasible for 290 prostatectomies. Since we do concur with the reviewers that ablating the effect of registration would be a valuable experiment, we have decided to make our code publicly available upon publication to stimulate method comparisons.

The second shared comment revealed some unclarity on the exact implementation of our baseline methods compared to the proposed multimodal approach, which led to some reservations regarding the benefits of the proposed approach. We would like to emphasize that all methods use the exact same 3D ResNet-18 model, preprocessing strategy (i.e. prostate gland mask-based cropping) and hyperparameters (i.e. weight decay). The inclusion of the fully supervised and unimodal SSL baseline effectively ablates the effect of adding unlabelled mpMRI data (fully supervised vs. unimodal SSL) and adding histopathology data (unimodal SSL vs. multimodal SSL). Specifically, the latter comparison provides strong evidence for the added value of histopathology data under otherwise identical circumstances. We believe that the code release would further elucidate this, but we also plan to clarify this in the final manuscript.

The first reviewer accurately notes that the cross-modal supervision signal is derived from a small paired subset (290/11075 patients), but we would like to clarify that this paired subset is disjoint from the internal test set of 2000 patients. We thus believe that the substantial improvement on this large-scale test set is a sign of robustness and generalization beyond the paired cases. Furthermore, while we could foresee a further improvement by using a larger paired subset, we believe the effectiveness at a smaller scale is an advantage of our method since it enables strong performance gains with comparatively limited data collection efforts. On a separate note, we respectfully note that the listed limitations, while fair, do not appear to align with a ‘major flaws’ (rating 1) assessment.

The second reviewer raises a valuable observation regarding bias in the prostatectomy specimens, since these cases are typically associated with more aggressive cancers. However, we should note that most of our development set (5257/7016 patients) comprises clinically insignificant prostate cancer, and one could argue that the model would benefit from seeing more of the underrepresented high-grade cases to learn a more fine-grained distinction between these grades. Considering this nuance, we agree with the reviewer that reporting the ISUP distribution would aid the interpretation of the results, and we aim to incorporate this in the final manuscript.

The third reviewer raises an interesting question whether performance gains are driven by PRISM or the general framework. We believe that these performance gains are driven by our robust framework, since different slide-level pooling methods and foundation models deliver comparative performance gains (Table 3). We also concur with the reviewer that there is untapped potential for interesting interpretability experiments (e.g. on which cases does multimodal SSL outperform unimodal SSL and why), but we reserved this for future work due to the page limit.

We thank the reviewers and ACs for their consideration.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
    • The ablation study does not provide full validation of the proposed method.
    • Lacks implementation details and a head-to-head comparison.
  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Although there are remaining concerns, some have been addressed by the authors.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Reject

  • Please justify your recommendation.

    After rebuttal, this paper receives two accept recommendations and one reject recommendation. The AC acknowledges that the paper addresses an important problem by using histopathology-derived supervision to improve mpMRI-based prostate cancer grading without requiring registration, and the large-scale evaluation shows promising gains over supervised and unimodal SSL baselines. However, the rebuttal does not sufficiently resolve key concerns about whether the observed improvements are truly attributable to latent histopathology supervision, given the limited paired pathology subset, lack of direct comparison with registration-based or alternative multimodal approaches, and incomplete ablation of cross-modal design choices. Important issues regarding prostatectomy-related bias, class imbalance, external validation, and threshold-dependent csPCa metrics also remain insufficiently addressed. Therefore, despite the practical motivation and promising empirical results, the AC believes that the current evidence is not yet strong enough to support the central claims and recommends rejection.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    This work addresses mpMRI prostate cancer classification by utilizing cross-modal latent alignment with whole slide images. The framework demonstrates strong resilience to data asymmetry, delivering notable diagnostic classification gains across an internal validation set while using only a 3% paired data ratio for training. The method is novel and can potentially generate high impact.



back to top