Abstract

Structured clinical report generation for 3D CT requires reasoning over long visual contexts while grounding predictions in lesion-relevant evidence. In practice, multimodal large language models (MLLM) often suffer from attention dispersion under long visual sequences, where critical cues are overwhelmed by background tokens, especially in low-data and imbalanced settings. We present a lesion-aware multimodal framework that addresses this challenge through two complementary mechanisms. First, we introduce a lesion-guided attention loss that regularizes cross-modal attention during training to encourage focus on lesion-relevant evidence without altering the inference interface. Second, we propose a compact set of Global Evidence Slots (GES) that summarize long visual sequences via cross-attention pooling and act as an explicit evidence bottleneck injected into the MLLM input, facilitating task-dependent routing of visual evidence. Together, these components form a hierarchical evidence token interface that combines fine-grained image tokens with compact evidence slots for robust lesion-centric reasoning. We evaluate our method on an ovarian tumor CT cohort with 12 structured attributes and validate its generalization on LIDC-IDRI. Experiments demonstrate consistent improvements over strong baselines.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1990_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{ZhaXiu_LesionCentric_MICCAI2026,
        author = { Zhao, Xiuheng AND Shi, Yiming AND Zhu, Xun AND Wang, Chen AND Cong, Fuze AND Yang, Jingzhi AND Wu, Ji AND He, Yonglan AND Li, Miao},
        title = { { Lesion-Centric Structured Report Generation for 3D CT with Multimodal Large Language Models via Hierarchical Evidence Tokens } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16879},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes a lesion-centric MLLM framework for structured attribute prediction from 3D CT. It introduces a lesion-guided attention loss to regularize cross-modal attention and Global Evidence Slots to summarize long visual sequences into compact evidence tokens for hierarchical lesion-aware reasoning.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses a meaningful problem in 3D CT, where lesion-relevant evidence is localized and long visual sequences can dilute important cues. 2.The proposed method is clean and well structured. The lesion-guided attention loss and GES target complementary aspects of the problem. 3.The experiments are reasonably complete, including backbone comparisons, lesion-prior settings, ablations, low-data evaluation, external validation on LIDC-IDRI, and intervention analysis on GES. 4.The empirical gains are substantial, especially on the ovarian cohort, and the ablations support the contribution of both proposed components.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The methodological novelty is moderate, as both lesion-aware attention regularization and compact latent/query-based evidence summarization are related to existing ideas, and the main contribution lies in their integration for 3D CT MLLMs. 2.A notable part of the gains may come from the lesion-prior design itself, including MaskZ slice selection and explicit lesion marking, so the incremental value of the hierarchical evidence interface could be discussed more clearly. 3.Some medical dataset and protocol details are still limited, including cohort split details, attribute definitions, handling of inapplicable labels, and LIDC-IDRI benchmark construction. 4.The ovarian cohort is still modest in size for a 12-task imbalanced structured prediction problem, making robustness on rare attributes harder to assess.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper presents a coherent lesion-centric MLLM framework for structured attribute prediction from 3D CT and provides a reasonably complete experimental validation. The motivation is clear, the method is clean, and the empirical gains are convincing across backbone comparisons, ablations, low-data settings, and external validation. I do think the methodological novelty is moderate and that some medical dataset details could be more rigorous. Nevertheless, the paper is overall above the acceptance threshold and represents a solid contribution in this application setting.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    1.The authors developed a lesion-aware framework featuring a hierarchical evidence token interface that integrates fine-grained image tokens with compact evidence slots for robust 3D reasoning. 2.This paper proposes a lesion-guided attention loss as a training-time regularizer to focus cross-modal attention on relevant evidence and stabilize optimization in imbalanced data regimes. 3.This paper introduces Global Evidence Slots to utilize cross-attention pooling to create an explicit evidence bottleneck, facilitating task-dependent routing of critical visual cues.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.This paper introduces a novel hierarchical interface that combines fine-grained image tokens with a compact “Global Evidence Slots” bottleneck to efficiently summarize long 3D CT sequences for MLLM reasoning. 2.This paper proposes an original lesion-guided attention loss that acts as a training-time regularizer to suppress background noise and focus cross-modal attention on critical diagnostic regions without increasing inference complexity. 3.The experimental results demonstrate significant clinical feasibility by achieving substantial performance gains over general-domain and medical-domain baselines on a real-world imbalanced ovarian tumor cohort and a public lung nodule benchmark.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The performance of the proposed method is highly dependent on the provision of precise 3D lesion masks, which may limit its real-world clinical utility where such manual segmentations are not always available or are subject to inter-observer variability. 2.While the Global Evidence Slots provide significant gains , the concept of using learnable queries and cross-attention to summarize long sequences draws heavily from existing architectures like the Perceiver and Slot Attention, reducing the purely structural novelty of the evidence bottleneck. 3.Although the lesion-guided attention loss does not affect inference speed, calculating this regularization term for every text-to-image token interaction during training could increase computational costs and memory requirements compared to standard fine-tuning.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper is technically sound and the experimental validation is rigorous, but these concerns regarding clinical practicality and limited theoretical departure from prior art led to the “Weak Reject” score. The framework’s fundamental performance depends on high-quality, 3D lesion masks, which may limit its practical value in clinical settings where manual segmentation is labor-intensive and often unavailable. Moreover, while the system is effective, the core components like the evidence bottleneck and cross-attention pooling are largely adaptations of established techniques from prior models. Although the paper promises “report generation,” the evaluation is primarily focused on discrete categorical attribute prediction, leaving the transition to nuanced, natural language clinical reporting largely unexplored.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    In their work, the authors propose methods to guide vision-language models by lesions to improve structured report generation of 3D CT scans. For this purpose, the authors introduce three measures: lesion-guided masking/marking, lesion-guided attention loss, and global evidence slots (GES). The GES are populated with relevant information from the 3D volume using cross-attention with a learned query matrix. They show that reducing the amount of background tokens using these measures improves the performance across multiple architectures. They validate their method through ablation and show that the GES contain relevant information that the models use during inference. They report performance on two datasets, a private and a public one.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Improvement over state of the art: The authors show a clear improvement in the structured report generation over state of the art.
    • Applicability: The adaptations the authors propose could be applied to a wide variety of MLLMs, since they mostly extend the models. The GES could also be transferred to other 3D imaging modalities or VLM architectures and could therefore be relevant to other domains.
    • Evaluation with multiple architectures: The authors evaluate their method on multiple architectures and show improvements in all settings.
    • Ablations: The authors show the improvements through ablation. This includes multiple architectures and two different datasets. With the ablation, they show consistent improvements.
    • Utility for low data settings: The authors show that the GES and lesion-guided attention loss improve the model performance in low-data regimes, which is relevant for many clinical applications.
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Segmentation mask guidance limits clinical transferability: The author’s approaches use masking and marking of lesions/nodules and a lesion/nodule guided attention. Those rely on existing segmentations. In their evaluation, the authors use high-quality annotations from the datasets. In cases with fully automated processing of the CT scans, these annotations might be sparse or unavailable, or would be provided by a segmentation model, potentially lowering the quality. The evaluation does not show how the approach performs in such cases, making it less expressive. In the introduction, the authors reference segmentation models as a reason why the masks are available.
    • GES Evaluation after Masking: The GES are only evaluated after applying the lesion mask and, therefore, already filtering a significant amount of background tokens. The evaluation does not allow any conclusions on how well the GES can summarize the evidence in low information density settings.
    • Limited evaluation of the multi-label classifier: Without any adaptations, the multi-label classifier outperforms all backbones except MedM-VL in the ablation of the lesion marking, but it is not included in any of the other ablations. The performance of the classifier makes it questionable whether an MLLM is necessary to generate a structured report from 3D volumes, especially in single-task settings.
    • The complete evaluation is only performed on the private dataset and not on LIDC-IDRI, limiting insights into the transfer of the improvements to other tasks.
    • Reproducibility: The description of the architecture is not detailed enough to reproduce the results without additional details (optimizer, loss weights, decoding strategy, seeds, LoRA settings). Furthermore, neither the code nor one of the training datasets is provided.
    • Dataset contents: The authors train and evaluate their model on one private dataset. While they briefly describe the dataset, details on the collection and annotation process are missing.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • References: References to the backbones (Qwen2.5-VL, M3D, InternVL3.5) are missing and should be provided, as they are a key part of the work.
    • Figure/Table order: The order of the figures and tables does not reflect their order in Section 3.5, hindering the flow of reading
    • The number of global evidence tokens in the experiments should be mentioned before the main results section (likely in 3.2 or 3.4)
    • Sometimes when the word “tokens” is used, it is not strictly clear whether vision or text tokens are meant.
    • Figure 1: The left side of the figure has no clear reading direction, making it difficult to understand. I think it would be clearer to understand if the text branch would also connect to the LLM from the top, since it is also an input. This way, input, output, and attention input would be more clearly separated.
    • Attention Loss and GES Evaluation: The authors do not include the InternVL3.5 model in the attention loss and GES evaluation, even though it outperformed Qwen and M3D in the previous settings. Additionally, the GES are not evaluated for Qwen and Intern3.5.- Number of GES: The number of GES is not ablated. It is not clear how the authors selected the number. From Figure 2 b, it seems that 3 slots are not attended to a lot and could therefore be removed.
    • Figures 1 and 2a mention LoRA, but the implementation details and Table 1 do not give any details on which models were trained with it.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The method shows clear improvement over the state of the art. Especially the GES could be useful for VLMs processing other 3D imaging modalities. While some of the weaknesses include the evaluation, it is quite thorough and clearly shows the advantages of the author’s method.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers and meta-reviewer for their constructive comments and for recognizing the promise of our lesion-centric framework for improving the reliability of 3D CT MLLMs. We would like to clarify the following points, especially the mask-practicability concern highlighted in the meta-review. Scope and reliance on lesion masks. We agree that requiring segmentation masks may limit direct deployment if high-quality manual masks are assumed. Our work is positioned in a lesion-prior setting: given a CT volume and a lesion/nodule prior, the goal is to study how such priors can be more effectively used for structured attribute reasoning, rather than to solve detection or segmentation. Importantly, our use of masks does not require high-fidelity boundary segmentation. MaskZ mainly needs to identify lesion-containing slices; explicit marking uses a minimum enclosing circle as a coarse localization cue; and the attention regularizer guides attention toward lesion-relevant regions rather than exact contours. Thus, the required prior is conceptually closer to coarse lesion localization, such as loose/interactive annotations or outputs from automatic segmentation, than to precise manual segmentation. We did not systematically evaluate imperfect or coarse masks in the submitted manuscript, so we will not claim verified robustness. We will instead clarify this scope and explicitly discuss coarse-mask/automatic-mask evaluation as an important future direction in the camera-ready version. Incremental value beyond MaskZ/marking. We also clarify that the reported gains are not solely due to lesion-prior preprocessing. All component analyses are conducted on top of the strongest MaskZ+mark baseline. Under this same setting, attention guidance further improves the baseline, GES yields larger gains, and their combination performs best. Moreover, zeroing or shuffling GES causes marked drops, supporting that the improvement comes from meaningful evidence slots rather than merely adding parameters or sequence length. Novelty of GES and relation to prior query/slot-based methods. We acknowledge that learnable queries and cross-attention pooling are related to Perceiver/Slot Attention-style representations, as cited in the paper. Our contribution is not claiming these primitives as new in isolation, but adapting them into a hierarchical evidence token interface for 3D CT MLLMs: fine-grained image tokens are preserved, while compact Global Evidence Slots are injected into the MLLM sequence and participate in language self-attention for task-dependent evidence routing. This differs from using query tokens only as a visual compression module. Structured report generation formulation. We appreciate the comment that our evaluation focuses on discrete attribute prediction. We will make the terminology clearer: this paper studies structured report generation in the form of standardized attribute-level entries, where each clinical field is queried and generated as a categorical answer. We do not claim to generate full free-form narrative radiology reports. Dataset, protocol, and reproducibility details. We agree that several details can be clearer. In the camera-ready version, we will add concise information on cohort split, annotation/label construction, conditional N.A. labels, LIDC-IDRI construction, number of GES, LoRA settings, loss weight, decoding strategy, optimizer/seeds, and missing backbone references. The private ovarian CT cohort cannot be publicly released due to clinical privacy constraints, but the LIDC-IDRI validation is built from a public dataset and will be described more explicitly. We appreciate the reviewers’ suggestions for additional analyses, including coarse/imperfect masks, GES-number ablations, and broader classifier/MLLM comparisons. Following the rebuttal guidelines, we do not introduce new experimental results here; we will instead improve the clarity of existing results and discuss these directions as future work.




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The proposed method targets at the attention dispersion and data imbalance problem of MLLM applications in report generation. The contributions of 3D image understanding with hierarchical evidence token “routing” appear promising in improving the reliability. However, as all reviewers pointed out, the requirement of segmentation mask might severely limit its practicability. The authors are encouraged to provide analysis (ablation if better, e.g., replacement of segmentation mask with looser, coarse-grained labeling/annotation as in interactive segmentation) and respond to the reviewers’ concerns.



back to top