List of Papers Browse by Subject Areas Author List
Abstract
The allocation of visual attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human attention helps medical imaging tasks such as classification and segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human attention into the training of pathologist report generation models. To this end, we collected a multimodal human-attention dataset of 121 prostate WSIs annotated with pathologists’ multi-scale viewport trajectories synchronized with the pathologists’ verbal descriptions and cursor movements for five clinically relevant components. We then fine-tune two report generation models with an attention-alignment loss that regularizes the model attention over image patches to match the distribution of pathologist attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal attention mechanisms (i.e., how image tokens are integrated into the language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output. Dataset and Code are available at: https://github.com/cvlab-stonybrook/Pathologist-Attention-Aligned-Report-Generation.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4588_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/cvlab-stonybrook/Pathologist-Attention-Aligned-Report-Generation
Link to the Dataset(s)
https://github.com/cvlab-stonybrook/Pathologist-Attention-Aligned-Report-Generation
BibTex
@InProceedings{XueRuo_Pathologist_MICCAI2026,
author = { Xue, Ruoyu AND Singh, Suryakant AND Chakraborty, Souradeep AND Marza, Pierre AND Yaskiv, Oksana AND Friedman, Constantin AND Sheuka, Natallia AND Friedman, Paul AND Ramlal, Bharat AND Knudsen, Beatrice AND Gupta, Rajarsi AND Saltz, Joel AND Prasanna, Prateek AND Zelinsky, Gregory AND Samaras, Dimitris},
title = { { Pathologist Attention–Aligned Report Generation for Prostate Histopathology } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16891},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper concerns the generation of prostate cancer pathology reports with training based on real human pathologist attention for better performance.
The method was evaluated on a multimodal dataset reviewed by 6 pathologists with their viewport trajectories recorded along with synchronized verbal descriptions.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The results show improved report quality and an interesting match of pathologist attention and algorithmic attention.
The pathologist-attention supervision for pathology report generation is novel and shows improvement.
A nice feature of this work is interpretability since the attention support at inference time matches human attention.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
It would be interesting to see how this method generalizes to other domains.
Example errors produced would be valuable.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Definite novelty and the evaluation is convincing.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The main contribution of this work is the introduction of a human-attention-aligned training framework for pathology report generation, where pathologist visual attention is used as auxiliary supervision to guide how vision–language models attend to whole-slide images. In particular, the authors collect a multimodal dataset of prostate WSIs with synchronized viewport trajectories and verbal annotations, and propose a plug-and-play attention-alignment loss that regularizes model attention over image tokens to match the distribution of pathologist attention via a KL divergence term. This approach is designed to improve both report quality and interpretability, by grounding generated text in diagnostically relevant regions, and is demonstrated across two architectures (HistGen and WSI-LLaVA) with consistent gains in NLP metrics, clinically relevant component accuracy, and alignment between model and human attention.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
I think this work tackles a clinically meaningful problem, namely how to better ground pathology report generation in diagnostically relevant regions of whole-slide images. The idea of leveraging pathologist attention as a proxy for clinical reasoning is compelling, as it directly connects model behavior to how experts actually examine slides. This is particularly important in pathology, where the diagnostic process is inherently spatial and sequential. The paper is also strong in that it goes beyond standard report generation and explicitly targets interpretability and visual evidence, which are critical for clinical trust and adoption. One of the major strengths of the work to me is the collection of a novel multimodal human-attention dataset. The dataset includes multi-scale viewport trajectories, cursor movements, and synchronized verbal descriptions across 121 prostate WSIs, which is non-trivial to acquire and represents a valuable contribution on its own. I think this is particularly interesting because it captures dynamic examination behavior, rather than static annotations such as ROIs or segmentation masks. Compared to prior work that uses weak supervision or heuristic region labels, this work offers a more realistic and fine-grained signal of what pathologists actually consider important during diagnosis. Another key strength is the simplicity and generality of the proposed attention-alignment formulation. The method introduces a relatively lightweight KL-divergence loss that aligns model attention with human attention distributions over image tokens, and can be applied to different architectures (both self-attention and cross-attention fusion). I think this is solid because it is plug-and-play, adds minimal computational overhead, and does not require attention signals at inference time. While attention supervision has been explored in other contexts, its application to pathology report generation with explicit human attention alignment is novel and well-motivated. Moreover, the current experimental evaluation is reasonably comprehensive and aligned with the claims. This work evaluates both report quality (BLEU, ROUGE, METEOR, and clinically relevant component accuracy) and visual grounding (NSS, KL divergence, ROC-AUC with segmentation), which provides a more complete picture of performance. The inclusion of two different model architectures (HistGen and WSI-LLaVA) strengthens the claim of general applicability. In addition, the qualitative results (Figures 3 and 4) clearly illustrate that the aligned models focus more on diagnostically relevant regions, which supports the interpretability argument beyond quantitative metrics.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
This work measures alignment between model and human attention using metrics such as NSS and KL divergence, and compares attention maps to segmentation labels (Table 2 at PAge 6), but does not include any “pathologist-in-the-loop” evaluation to confirm that the highlighted regions are clinically meaningful or useful for decision-making. Given that the central claim is improved interpretability through alignment with expert attention, this is an important missing piece. Also, the core idea, such as aligning model attention with human attention via a KL-divergence loss is conceptually straightforward and closely related to existing attention supervision and gaze-alignment approaches. For example, prior work such as GazeVLM (https://arxiv.org/abs/2510.21356) employs a similar KL-based alignment between human gaze and model attention, and attention-guided learning has also been explored in medical imaging for classification and segmentation tasks. The main novelty here is the application to pathology report generation, but the underlying formulation is not fundamentally new. I think it would benefit from clarifying what is uniquely enabled in this setting beyond prior attention supervision approaches. The second limitation to me is the scale and scope of the dataset, which is relatively small (121 WSIs, ~7 hours of annotation) and limited to a single cancer type (prostate) and a small number of pathologists (six). While the dataset collection is a strength, it also raises concerns about generalizability and variability in expert behavior. Pathologist attention can vary significantly across individuals and expertise levels, and the paper does not analyze inter-observer variability or whether the model is learning a consistent signal versus overfitting to specific annotators. In addition, only a subset of annotated components is used due to sparsity, which further limits the coverage of clinical concepts. While improvements in NLP metrics and component-level accuracy are reported, it is not entirely clear whether these gains are directly attributable to better visual grounding or to regularization effects in a low-data setting (as the authors themselves suggest for WSI-LLaVA). Moreover, the reliance on automated metrics and LLM-based extraction of clinical components (e.g., using ChatGPT for evaluation) introduces additional uncertainty. The lack of uncertainty-quantification is another concern.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This contribution copes with an important and clinically relevant problem in healthcare, grounding pathology report generation in meaningful visual evidence, and introduces a practical and well-motivated approach by incorporating pathologist attention as supervision. The dataset collection, although limited in scale, is a meaningful contribution, and the method is simple, general, and shows consistent improvements across two architectures in both report quality and attention alignment. That being said, the methodological novelty is somehow limited, as the core idea of attention alignment via KL divergence is closely related to prior gaze-supervised approaches. In addition, the evaluation of interpretability is indirect and lacks domain-expert validation, and the dataset is relatively small and limited to a single cancer type. Some of the claims around interpretability and clinical relevance could also be more carefully calibrated.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I appreciate authors’ careful feedback. The clarification that the primary contribution lies not in a new attention mechanism itself, but rather in the end-to-end integration of pathologist attention into pathology report generation, along with the associated multimodal dataset and grounding pipeline, makes the current contribution more coherent and better aligned with the presented evidence. I also think the additional analyses improve the paper meaningfully. Most importantly, the authors addressed my concern regarding interpretability evaluation by including pathologist-involved assessment of the generated visual support. While this evaluation is relatively limited in scale, the finding that pathologists preferred the aligned visual support in 80% of cases provides stronger evidence that the attention maps are clinically meaningful rather than merely visually plausible. The additional Gleason classification analysis, inter-reader attention analysis, and discussion of failure cases also strengthen the practical relevance of the work. At the same time, my earlier concerns are not fully addressed. The methodological novelty of the alignment formulation itself remains somewhat limited, and the dataset is still relatively small and restricted to a single cancer type. In addition, the broader generalization claims remain preliminary. However, I think the current rebuttal adequately clarifies the intended scope of the manuscript and supports the contribution as a practically useful and clinically motivated system-level advance rather than a fundamentally new attention-learning formulation. Thank you!
Review #3
- Please describe the contribution of the paper
The authors proposed a new framework: Align pre-trained pathology LLMs’ visual attention with the human pathologist’s attention, which could improve the model’s performance. It essentially provide further supervision signal for the model to be supervised fine tune. The authors validated on prostate pathology datasets with two models and two different attention mechanisms, helping improve the performance on the report generation and visual question answering.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The strength of this paper is the innovative way to create the human attention map and how it can be incorporated into model training.
The paper does a good job of evaluating their framework on both different models and, more importantly, on different attention mechanisms.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.I do not think this is a plug-and-play framework that the author stated. It requires deliberately finetuning. Decisions like which layers to be finetuned and also the coefficient of the loss component. According to different pre-trained models, they have different visual token processing, which should have different adaptations of the attention map.
2.Will the data be published, since it’s one of the major contributions mentioned?
3.Lack of translations. The authors did not specify what the final task is: whether to improve the ability of pathologies MLLM in general or to improve specific cancer types, like the prostate, which is the dataset used. If it’s general, which still require huge amount of different cancer types’ annotation from human pathologists. If it’s for a specific task, what is the advantage of a human attention map over other types of human feedback that are more explicit, like the direct semantic labels for the pathology images, segmentation mask of the ROI.
4.The visual support cannot prove that the interpretability is enhanced. This last decoder layer is always available with or without this framework.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1.The ultimate contribution of this work to the pathology Mutimodal report generation model is not clear. The extendability is weak based on the current narrative.
2.The three main contributions the authors stated are not well supported by evidence.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the Meta-Reviewer (MR) and Reviewers 1, 2, and 3 (R1, R2, R3) for their insightful feedback. We organize our responses as (1)–(11), with “<>” indicating the corresponding reviewer(s).
(1) <R1, R2, R3, MR> If accepted, we will release our code and dataset.
(2)
ACC-G for clinical utility: After additional analysis on the performance of the model on gleason score classification (CLS) following HistGen’s CLS protocol, Gleason CLS AUC improves from 0.71 (baseline) to 0.77 (Ours). It is higher than ACC-G since ACC-G is extracted from generated reports, where the model is optimized for free-text generation rather than score prediction. But the gains over baselines and the pathologist-involved analysis in Response (9) support its clinical utility. (3)
Generalization: Our observation on PathoGaze [1], a breast-cancer attention dataset with classification (CLS) labels, suggests that human-attention alignment can also improve CLS on another domain from 0.833 to 0.989. (4) Error Analysis: For small tumor regions where the baseline VLM is strongly mislocalized, alignment provides only partial correction as it remains constrained by the VLM backbone. (5)
Methodological novelty: While we acknowledge GazeVLM in Sec. 3.2, our primary contribution is the novel end-to-end pipeline for pathology report generation. We define a new research task and provide a proof-of-concept for grounding clinical narratives in pathologist attention. (6)
Component coverage: We excluded rare components such as seminal vesicle invasion (positive in only 2 slides), that are insufficient for reliable attention supervision and evaluation. (7)
Influence of personalization: While our six pathologists (2 specialists, 2 generalists, 2 residents) show diverse gaze behaviors, our "positive-component-filtering" distills shared diagnostic evidence from individual attention. For multi-reader slides, this approach increases heatmap similarity from 0.55 to 0.71. (8) Regularization effect and uncertainty: Gains in Tab. 2 suggest that improvements stem from visual grounding rather than mere text-side regularization. Manual verification showed 100% agreement with the LLM results due to the structured reports. (9) <R2, R3> Evaluation of Interpretability: Our goal is not to add a new visual-support output, but to make existing attention maps more clinically useful, as shown in Tab 2.Pathologists’ evaluation confirms that our method provides strong tumor-focused and clinically meaningful visual support, preferred in 80% of cases over baselines on overall quality and component-specific diagnosis.
(10)
“Plug-and-Play”: Our method requires no architectural changes and adds no parameters. The finetuned layers and original loss coefficients follow each backbone’s default design, while the KL-loss coefficient is shared across models. The only model-specific difference is dataset preprocessing, i.e., grid alignment between gaze heatmaps and visual tokens. Using last-layer attention follows standard grounding practices [2,3] and consistently achieves the best. (11)
Lack of translation: We focus on prostate cancer report generation, which adapts pre-trained pathology VLMs to a clinical domain. While human attention provides efficient, diagnosis-aware supervision (3 min/slide), segmentation masks are 10-times more costly and frequently unavailable for report-level components like surgical margins. Also, synthetic masks add noise due to current Gleason classification limitations, discussed in Response (2). We expect strong transfer properties to novel cancers once attention data is scaled (discussed in Response (3)), with future work focused on broader cross-domain generalization. References: [1] Eye-Tracking, Mouse Tracking, Stimulus Tracking,and Decision-Making Datasets in Digital Pathology. [2] Grounded Language-Image Pre-training. [3] Pointing and Asking: A Context-and-Task-Aware Framework for Visual Question Answering.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This is an elegant approach to include the gaze information from different pathologists, showing how it helps improve report generation. Its a nice study, but I invite the authors to consider using aditional approaches for baseline, and evaluation in other independent datasets, maybe where reports may not be available but ground Grade groups are, like prior pathology grading challanges. The ACC-G to me is on the low side for clinical utility, but maybe with additional evidence, examples and error analysis we can better appreciate the value of the approach.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Congratulations for your submission. Well done.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The reviewers were largely positive following the rebuttal. The most engaged reviewer noted that the authors clarified the paper’s core contribution. I find the rebuttal addresses the main concerns adequately. Two reviewers lean toward acceptance, and the reviewer inclined to reject did not submit post-rebuttal comments. On balance, the rebuttal is convincing and the contribution is sound, so I recommend acceptance.
