Abstract

Cancer prognosis and recurrence prediction from histopathology images has relied on morphology focused models or genomic data. However, these approaches insufficiently capture the spatial organization and interactions between epithelial, stromal, and immune cell populations, increasingly recognized as important contributors to disease progression. We propose Deep-IMMA (Deep Immune-Aware Multimodal Multi-task Alignment), a general multimodal learning framework that derives biologically grounded prognostic representations directly from histopathology images. Deep-IMMA introduces a two-stage architecture that encodes the tumor–immune microenvironment via structured multimodal supervision. In Stage 1, tile-level visual embeddings are learned via joint multi-task multimodal supervision from structured textual captions and spatial cell-type maps, with an auxiliary cross-modal consistency objective, yielding semantically enriched and immune-aware representations. This joint training integrates vision, text, and spatial priors while requiring only histology image at inference time. In Stage 2, these representations are aggregated using a spatially aware Multiple Instance Learning module that explicitly preserves neighborhood relationships and immune topology, improving slide-level risk discrimination. Using prostate cancer as a case study, we evaluate Deep-IMMA on a tri-modal prostate cancer histology dataset (Prostate-TriMod), comprising 717 prostate tissue microarray (TMA) core images from 212 patients. Deep-IMMA achieves a test AUROC of 0.761 (95% CI: 0.666–0.847), outperforming prognostic models based on only morphology and clinical data for recurrence prediction. Importantly, Deep-IMMA generates interpretable spatial–textual outputs, including tile-level semantic descriptions, immune-infiltration and attention heatmaps, enabling spatially grounded risk interpretation. These results demonstrate that immune-aware multimodal supervision can establish a robust, transparent and cost-effective baseline for cancer prognosis, bypassing expensive auxiliary data.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5348_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/iamkj03/DEEP-IMMA

Link to the Dataset(s)

Prostate-TriMod: https://zenodo.org/records/18209419

BibTex

@InProceedings{JunKye_DeepIMMA_MICCAI2026,
        author = { Jung, Kyeong Joo AND Rout, Swapna Sourav AND Qiu, Jianwei AND Cho, Sanghee AND McDonough, Elizabeth AND Ghose, Soumya AND Chadwick, Chrystal AND West, Robert AND Brooks, James D. AND Ginty, Fiona AND Mallick, Parag AND Jadhav, Kshitij AND Machiraju, Raghu},
        title = { { Deep-IMMA: an Interpretable Immune-Aware Multimodal Deep Representation Learning for Cancer Prognosis and Recurrence Prediction } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16887},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes a 2 stage framework for predicting prostate cancer biochemical recurrence from histopathology slides by learning immune-aware representations. In the first stage, it trains a vision backbone using tissue maps with structured cell/immune annotations, and text captions to build biologically grounded tile-level embeddings. In the second stage, these embeddings are aggregated with a spatially aware multiple instance learning model that preserves spatial relationships and highlights prognostic regions. Their method is designed to improve prediction performance and interpretability by modeling tumor and immune interactions. They used a 212-patient cohort, and their reports improved recurrence prediction over standard baselines

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1) They have a strong biological motivation focused on tumor–immune microenvironment which is evident through explicit modeling of tissue compartments. 2) They have rich multi-task supervision (segmentation, counts, immune labels, captions) to enrich representations. 3) Their model produces interpretability attention maps of immune infiltration between recurrence and non-recurrence phenotypes. 4) They also combine vision, language, and spatial signals in a unified framework.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1) There’s a heavy reliance on expensive supervision, including pixel-level (on tiles) tissue segmentation maps. Additionally, the dice score is very low ~0.22.They argue segmentation is not meant to be precise but “interaction-centric,” but this is not fully convincing. If segmentation is inaccurate, it is unclear how reliable downstream biological supervision is. 2) it is unclear what the role of the Q-former is. While the attention maps are good, have they been evaluated by a pathologist? 3) No external validation: it would be great if this were validated. It is unclear whether the claimed tumor–immune interaction is actually modeled or if the model only learns co-occurrence patterns through shared supervision and spatial proximity. 4) It is unclear what staining or slide type was used. Is it H&E primarily? Why does it say virtual H&E from DAPI? The images seem to be TMA and not biopsy slides. 5) Usually prognostic models are evaluated in terms of C-index, hazard ratio, survival curves. 6) Also unclear where is the structured text from captions coming from?

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Modeling of biological phenotypes from imaging and text input for recurrence prediction is interesting. But missing dataset details, staining, interpretability of immune infiltration, prognostic modeling has reduced enthusiasm.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    Deep-IMMA is a two-stage foundation model adaptation framework that learns biologically grounded prognostic representations from histopathology by explicitly encoding tumor–immune microenvironment (TIME) structure through structured multi-task supervision. Stage 1 adapts a frozen UNI backbone with a Q‑Former bottleneck using uncertainty‑weighted multi‑task learning to jointly model tissue architecture, immune infiltration phenotypes, cellular composition, and vision–text alignment with BioMedBERT, producing immune‑aware tile embeddings. Stage 2 aggregates spatially encoded tiles with a Top‑K gated MIL model to predict biochemical recurrence with auxiliary Gleason grading and attention sparsity. Extensive ablations confirm each component’s necessity, and on a tri‑modal prostate cancer cohort the full model achieves AUROC 0.761, outperforming ABMIL and logistic baselines while yielding interpretable tissue, immune, and attention maps.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Biologically grounded supervision is a compelling idea. Instead of purely morphological features, the framework explicitly encodes immune-tumor spatial interactions (6 immune infiltration phenotypes) as structured supervision signals. This is well-motivated by pathology literature showing immune contexture predicts outcomes. 2.Unified multi-task framework is elegant. Jointly training segmentation, captioning, classification, and regression within a single Stage 1 pipeline with uncertainty-weighted losses is a clean design that avoids training separate models. 3.Thorough ablation study. The paper systematically ablates Stage 1 components (backbone, alignment loss) and Stage 2 components (spatial encoding, Top-K, auxiliary grade, sparsity regularization), with CIs for all results. 5.Practical clinical value. The model works from histology alone (no MRI/genomics), making it more clinically accessible.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.very small dataset and single-site evaluation. Only 212 patients / 711 slides from a single institution. The 95% CI for AUROC is wide (0.666-0.847), and with only 44 test patients, results could be unstable. No external validation on any public prostate cancer dataset (e.g., TCGA-PRAD) is provided. For clinical prognosis claims, multi-site validation is essential. 2.Limited baselines — no comparison with modern pathology VLMs or survival models. The baselines are only Logistic Regression and ABMIL. No comparison with CONCH, PathChat, CLAM, TransMIL, or any recent WSI-level prognosis methods. The claim of outperforming “morphology-only” models lacks comparison with current SOTA slide-level prognosis methods that also use foundation model features. 3.Virtual H&E images (from DAPI/autofluorescence) limit generalizability. The entire pipeline uses virtual H&E converted from fluorescence imaging, not standard H&E. This is a significant limitation for clinical deployment since most pathology labs use conventional H&E staining. The paper does not discuss or validate whether the learned representations transfer to real H&E. 4.U-net decoder architecture is underspecified. Critically, the paper does not specify which intermediate ViT layers from the frozen UNI backbone provide skip connections to the U-net decoder, how many skip connections are used, or the decoder architecture details (layers, channels). Or do you directly use Qformer semantic ebmeddings for segmentation? how do you do that? Since standard ViTs lack the natural multi-scale feature hierarchies of CNNs, how skip connections are constructed for a U-net decoder from a ViT is a non-trivial design choice that directly impacts segmentation quality — and this is entirely undocumented, hindering reproducibility. 5.No survival analysis or time-to-event modeling — binary BCR classification discards censored data. BCR prediction is framed as binary classification rather than survival analysis (Cox regression, C-index), which is the standard approach for recurrence prediction in oncology. This is a significant methodological limitation: the paper mentions excluding slides that “lacked BCR labels,” but in clinical practice many patients are censored (lost to follow-up or have not yet recurred at the time of data collection). Binary classification forces exclusion of censored patients, wasting valuable data and potentially introducing selection bias. Survival analysis methods (e.g., Cox proportional hazards, discrete-time survival models) naturally handle censoring and provide time-to-event predictions that are more clinically actionable than binary yes/no recurrence labels. 6.Batch size for Stage 1 multi-task training is not reported — a critical omission for the InfoNCE alignment loss. The paper does not report the training batch size for Stage 1.While the segmentation, classification, and regression losses work well with small batches, the InfoNCE contrastive alignment loss (L_align) is known to be highly sensitive to batch size. Without knowing the batch size, it is impossible to assess whether the contrastive alignment is properly trained or undertrained, and the result cannot be reproduced.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The multi-task semantic representaiton learning is innovative and fit the biological understanding of the problem. However, they trained the model on virtual HE but miss an important validation on real HE data. In addition, the baseline model they pick is too limited.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper introduces a multi-task multimodal training framework that incorporates immune-related supervision to learn interpretable and biologically grounded representations for recurrence prediction.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The focus on immune-aware representation learning is well-motivated and clinically relevant. 2.The use of structured supervision provides a biologically grounded learning signal. 3.The framework produces interpretable outputs, including attention maps and immune patterns. 4.The two-stage design is reasonable and supported by ablation studies.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The methodological novelty is moderate. The framework primarily integrates existing components. 2.The multi-task objective is complex, and the relative contribution of individual supervision signals is not fully analyzed. 3.Baseline comparisons could be stronger, particularly with more recent pathology foundation models. 4.Interpretability claims are mostly qualitative and would benefit from more rigorous validation.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses an important problem and proposes a biologically grounded framework that integrates multiple supervision signals for improved representation learning. While the methodological novelty is moderate and the evaluation is somewhat limited, the approach is well-motivated and demonstrates consistent improvements, making it a solid contribution to the field.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers and Area Chair for the constructive feedback. Below, we clarify points that may have led to ambiguity in the manuscript. 1.Implementation Details and Architecture (R1, R2, R3): Stage 1 of our model uses the UNI vision transformer backbone. For segmentation, visual tokens from blocks 3, 6, and 9 are reshaped into spatial maps and projected to 256, 128, and 64 channels, respectively, before being used as decoder skip features. The Q-Former is a compact semantic bottleneck for vision-text alignment: 64 learnable queries attend over image tokens, and the query features are used for auxiliary prediction, image-text alignment, and downstream aggregation. Batch sizes were 32 (Stage 1) and 4 (Stage 2). 2.Data Representation, Captions, and Domain Gap (R1, R2, MR): The image data consist of prostate tissue microarray (TMA) cores rather than whole biopsy-slide WSIs. Each core includes virtual H&E/histology images generated from DAPI/autofluorescence imaging, paired single-cell-derived tissue maps, immune phenotypes, and structured captions. Captions are rule-based summaries from the single-cell table, including cell counts, cell-type abundance levels, and immune-infiltration phenotypes; they are not free-form manual annotations or generated text. Regarding the virtual H&E domain gap, Deep-IMMA builds on UNI, pretrained on large-scale real H&E WSIs, including prostate pathology data. We agree that real H&E validation would further strengthen generalizability. 3.Survival Analysis and Risk Stratification (R1, R2, MR): We acknowledge that time-to-event modeling and C-index evaluation are clinically important for predicting recurrence, especially in the presence of censoring. In this study, recurrence prediction was formulated as binary BCR classification to evaluate immune-aware representation learning under a controlled patient-level split. Stage 2 MIL model outputs a continuous recurrence probability, interpretable as a risk score for stratifying recurrence phenotypes. Thus, current results demonstrate risk-discrimination capability, while survival modeling with censoring is outside the scope of the current project. 4.Segmentation Dice and Biological Supervision Reliability (R1, R2): We agree that the segmentation branch should not be interpreted as a high-accuracy standalone segmentation model. Its role is an auxiliary spatial/semantic anchor for tissue composition and tumor–immune topology, not pixel-perfect boundaries. Structured captions and immune-phenotype labels are derived from single-cell/tissue annotations rather than from predicted segmentation maps. Thus, low boundary-level DICE values do not directly imply that immune supervision itself is unreliable. Ablations in Table 2 suggest that spatially structured supervision and aggregation contribute to discrimination of recurrence risk. 5.Interpretability (R3, MR): We present attention and immune-pattern maps as evidence. The maps highlight regions with immune-infiltration patterns consistent with known tumor–immune dynamics and help examine which tissue regions contribute to recurrence prediction. Pathologist assessment would further strengthen this interpretation. 6.Baselines Comparison and Contribution (R2, R3, MR) Broader benchmarking would further strengthen the study. The submitted experiments compare pathology foundation backbones, including UNI, Prov-GigaPath, and H-optimus-0, and evaluate recurrence prediction against logistic regression and ABMIL with targeted ablations (Table 1,2). These analyses were designed to isolate the effects of immune-structured supervision and spatial aggregation within the same patient-level split, rather than to provide exhaustive benchmarking. The main contribution is injecting tumor-relative immune topology into foundation-model adaptation, enabling H&E-only risk discrimination and spatial interpretation.




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This work proposes a multimodal, immune-aware framework for prostate cancer recurrence prediction that combines structured supervision signals, vision-language alignment, and spatial MIL aggregation to learn biologically informed representations from histopathology. Reviewers consistently acknowledge that the problem is clinically meaningful and that the attempt to explicitly model tumor–immune microenvironment interactions is a strong and relevant motivation. The overall pipeline is well-structured and demonstrates thoughtful integration of multiple supervision sources, with the potential to support more interpretable prognostic modeling. The use of histology-only inputs for inference is also noted as a practical advantage.

    At the same time, there is a shared concern that the methodological novelty of the work is limited. The proposed approach largely integrates existing components, including multitask learning, vision-language feature alignment, and MIL-based aggregation, into a unified pipeline. While this combination is reasonable and well-engineered, it is not yet fully clear what fundamentally new learning principle or architectural contribution is introduced beyond this integration.

    There are also concerns regarding the strength and clarity of the experimental validation. The study is conducted on a relatively small, single-institution cohort, which limits confidence in generalization. Several reviewers also note that the evaluation would benefit from stronger comparison against more recent pathology foundation models and more clinically standard prognostic frameworks. In addition, the use of a binary recurrence formulation instead of time-to-event survival modeling raises questions about clinical completeness, as it discards censoring information that is typically important in oncological prognosis.

    Another recurring point is the limited clarity around certain components of the framework. In particular, the role and construction of structured textual supervision, the reliability and interpretation of immune-related annotations, and the relative contribution of each supervision signal are not fully disentangled. This makes it difficult to assess how much of the reported performance gain is driven by biologically meaningful learning versus correlated supervision or model scaling effects. Interpretability claims based on attention maps are also viewed as promising but would benefit from more rigorous validation.

    Overall, the submission addresses an important clinical problem and presents a well-motivated multimodal learning framework with clear practical relevance. Despite limitations in novelty and validation breadth, the work is solid, clinically grounded, and of interest to the community. With improved clarity and stronger experimental support, it could represent a meaningful contribution.



back to top