Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Automatic and accurate segmentation of brain MR images throughout the human lifespan into tissue and structure is crucial for understanding brain development and diagnosing diseases. However, challenges arise from the intricate variations in brain appearance due to rapid early brain development, aging, and disorders, compounded by the limited availability of manually labeled datasets. In response, we present a two-step segmentation framework employing Knowledge-Guided Prompt Learning (KGPL) for brain MRI. Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal labels, followed by the incorporation of knowledge-driven embeddings learned from image-text alignment into the models. The introduction of knowledge-wise prompts captures semantic relationships between anatomical variability and biological processes, enabling models to learn structural feature embeddings across diverse age groups. Experimental findings demonstrate the superiority and robustness of our proposed method, particularly noticeable when employing Swin UNETR as the backbone. Our approach achieves average DSC values of 95.17% and 94.19% for brain tissue and structure segmentation, respectively. Our code is available at https://github.com/TL9792/KGPL.



Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1321_paper.pdf

SharedIt Link: https://rdcu.be/dV1Og

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_23

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1321_supp.pdf

Link to the Code Repository

https://github.com/TL9792/KGPL

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Ten_KnowledgeGuided_MICCAI2024,
        author = { Teng, Lin and Zhao, Zihao and Huang, Jiawei and Cao, Zehong and Meng, Runqi and Shi, Feng and Shen, Dinggang},
        title = { { Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15002},
        month = {October},
        page = {238 -- 248}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The author noticed that brain tissue segmentation is related with gender, age and health condition. To incorporate this information, the author proposed to incoprate the texts to the backbone network and demonstrate the effectiveness of including text embedding on different backbones.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    One interesting way to apply multi model.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Can author elaborate more about Model Fine-tuned Guided by Knowledge-wise Prompt? It is hard to follow and can author elaborate why it can save parameters at the same time?

    • It seems that information contained within text like gender, age and health condition can be represented by scalar values. This is one important baseline to demonstrate effectiveness of texts. Does author perform this experiment before?
    • This manuscript focused on the lifespan segmentation. Can author show more demographic information about training, validation and test cohort. Also, how does author handle longitudinal dataset?
    • Is that possible to show subject from different ages such as 0 ~ 3, 20 ~ 60 and 60 ~ 80 with different health condition and gender quantitatively and qualitatively?
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    The author does not mention open source code

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Please refer box 6.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper organization and results analysis

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The author has addressed my concerns.



Review #2

  • Please describe the contribution of the paper

    This article presents a two-step segmentation framework employing knowledge-guided prompt learning (KGPL) for brain MRI at various stages throughout the lifespan. The authors choose to pre-train segmentation models on large-scale datasets with sub-optimal labels, and they incorporate the knowledge-driven embeddings learned from image and text that are aligned from the model.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well structured, use new methods for segmentation such as prompting for a not well addressed question (segmentation of brain lifespan MRI), with a comparison with the litterature and the use of metrics.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Weaknesses are linked with the weak characterisation of the encoder and decoders architectures and the denomination of the Lifespan term.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    As mentionned in the paragraph 6, authors should specify some details about the encoder/decoder architectures.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    How many layers does the encoder/decoder architectures contain? Do they contain skip connections? How many parameters, memory and time does it take to run the entire process? No code is provided.

    Authors are using adolescent, adult and ederly brain images datasets. In the future, authors may include infants’ brain MRI to complete the use of their architecture and particularly correspond to the “lifespan” word they use.

    The authors may add annexes with other comparison of others brain structures.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper is clear and presents an architecture which could work to segment various brain structures with prompting on several datasets all accross the human lifespan excluding infants. The comparison is fair, but some details should be mentionned such as the contents of the encoder/decoder architecture to be more reproducible. Only the values from right and left hippocampus are written.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The author present a two-step framework with knowledge-guided prompt learning (KGPL) tailored for dense segmentation of brain MR images across diverse age groups.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    well organized paper and writing.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The author mentioned the MRI brain segmentation with whole lifespan. I think the author should provide detailed analysis for different age groups. For example, Figure 2 and Figure 3 should mention the subject’s age.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. Detailed analysis for different ages group especially for young people and adults, since the data ABCD is for 9-11 ages.
    2. In Figure 2, the last column is the ground truth, why is the red boundary not consistent? Please give explanations.
    3. Authors should provide supplementary results for all the regions segmentation results for tissue and structure segmentation.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Methodology, experiments.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #4

  • Please describe the contribution of the paper

    The paper offers an innovative approach for brain tissue and structure segmentation using knowledge guided prompt learning, to cover not only the intricate brain structure but also through different ages and diseases.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Including the textual information context for brain tissue / structure segmentation instead of only image based information using BiomedCLIP.
    • A huge reduction of model parameters and time consumed by focusing on additional knowledge embedding without sacrificing performance.
    • Extensive and detailed comparison of the proposed architecture with state of the art backbones.
    • Benefiting from a wide range of patient ages and diseases using multiple dataset repositories.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • Unclear about the source of the textual information for the knowledge driven embedding used in fine tuning.
  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Need more on the textual information collection and pre processing.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Good and clear graphical representation of the architecture. Great representation of the work without getting lost in details.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Proving the efficacy of textual context use in brain structure / tissue segmentation, smart use of knowledge embedding.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Strong Accept — must be accepted due to excellence (6)

  • [Post rebuttal] Please justify your decision

    An interesting approach and valuable insights regarding the use of Text embedding in segmentation task, also the authors successfully addressed reviewers concerns regarding the proposed architecture



Author Feedback

We thank AC and all reviewers for their insightful comments. Below are our point-by-point responses.

  • Technical details (R1 & R3) To ensure reproducibility and interpretability, we will release our code on Github, with some details also provided below. The backbones involve U-Net, UNETR, and Swin UNETR, with their original architectures used in our study, i.e., encoders have four CNN-based or twelve transformer-based blocks, decoders have four or five CNN-based blocks, and skip connections between the encoder and the decoder are included. The parameters, memory, and time for these backbones are (14.79M, 20879MiB, 4 days), (4.46M, 14931MiB, 3 days), and (54.47M, 16425MiB, 1 day), respectively. Our “Model Fine-tuned Guided by Knowledge-wise Prompt” approach incorporates a small set of tunable embeddings from the inherent knowledge of BiomedCLIP. These embeddings are warmed-started from subject’s textual information and concatenated with image embeddings [n, hidden_dim] along the n dimension as inputs, aiming to guide models for extracting fine-grained anatomical features. By only updating the learnable parameters and decoder while fixing the encoder, our method can be applied to multiple downstream datasets in a parameter-efficient manner. All these details will be included in the final paper.
  • Demographic analysis of datasets (R3 & R5) We categorize subjects into different age groups, including 795 adolescents (9-19), 755 adults (20-59), and 1467 elderly (60+) subjects. For pre-training, training/validation datasets consist of 413/64 adolescents, 392/61 adults, and 762/118 elderly subjects (mean age: 12.9/14.6, 36.8/41.4, and 78.2/77.3 years), respectively. For fine-tuning, training/validation/test cohorts consist of 254/32/32 adolescents, 242/30/30 adults, and 470/58/59 elderly (mean age: 14.7/15.3/14.8, 45.2/45.4/53.2, and 78.2/77.7/76.3 years), respectively. The gender distributions are 56%/62%/60% males and 44%/38%/40% females, respectively. And the disease distributions are 30%/10%/10%/20%/30%, 28%/14%/10%/15%/33%, and 24%/16%/13%/18%/29% for AD (Alzheimer’s Disease)/MCI (Mild Cognitive Impairment)/SMC (Subjective Memory Cognition)/ASD (Autism Spectrum Disorders)/HC (Health Condition), respectively. We will provide demographic information using pie charts in Supplementary Materials. The above split is for subjects at one-time point, while longitudinal subjects can only be included into the same cohort to prevent data leakage.
  • Detailed analysis of results (R1 & R3 & R5) Our method achieves superior performance in brain structure segmentation, and the improvement is statistically significant (p-value < 0.05) as demonstrated by paired t-test. Due to page constraints, we show segmentation results for the hippocampus. The complete comparison results of brain tissue and structure segmentations will be provided in Supplementary Materials. In Fig.2, the ground truth results shown in rows 1-3 and 4-6 are from two subjects, so the red boundaries are not aligned. We will describe the subject’s information in the captions of Figures 2 and 3.
  • Effectiveness of texts (R3) We have conducted an ablation study using one-hot encoding to represent texts. Compared with the case of using only image formation, the average DSC/ASD improved by 0.68% (93.81 to 94.49)/2.55% (26.02 to 23.47) and 0.67% (93.04 to 93.71)/3.10% (21.28 to 18.18) for brain tissue and structure segmentations, respectively. The results demonstrate that texts provide valuable semantic information and improve the performance of models. We will consider it as a baseline in the extension of this work.
  • Minor concerns (R1 & R4) We apologize for omitting the explanation of the lifespan term. Here, lifespan refers to the period between birth and death, emphasizing the wide age range covered in our study. All textual information comes from the corresponding data acquisition centers and their official websites. We will incorporate all this information in the final paper.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The paper presents an interesting approach with knowledge-wise prompt to enhance brain MRI segmentation. With the rebuttal, the authors have addressed the major concerns of the reviewers.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The paper presents an interesting approach with knowledge-wise prompt to enhance brain MRI segmentation. With the rebuttal, the authors have addressed the major concerns of the reviewers.



back to top