Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Using brain imaging data to predict the non-neuroimaging phenotypes at the individual level is a fundamental goal of system neuroscience. Despite its significance, the high acquisition cost of functional Magnetic Resonance Imaging (fMRI) hampers its clinical translation in phenotype prediction, while the analysis based solely on cost-efficient T1-weighted (T1w) MRI yields inferior performance than fMRI. The reasons lie in that existing works ignore two significant challenges. 1) they neglect the knowledge transfer from fMRI to T1w MRI, failing to achieve effective prediction using cost-efficient T1w MRI. 2) They are limited to predicting a single phenotype and cannot capture the intrinsic dependence among various phenotypes, such as strength and endurance, preventing comprehensive and accurate clinical analysis. To tackle these issues, we propose an FMRI to T1w MRI knowledge transfer Network (F2TNet) to achieve cost-efficient and effective analysis on brain multi-phenotype, representing the first attempt in this field, which consists of a Phenotypes-guided Knowledge Transfer (PgKT) module and a modality-aware Multi-phenotype Prediction (MpP) module. Specifically, PgKT aligns brain nodes across modalities by solving a bipartite graph-matching problem, thereby achieving adaptive knowledge transfer from fMRI to T1w MRI through the guidance of multi-phenotype. Then, MpP enriches the phenotype codes with cross-modal complementary information and decomposes these codes to enable accurate multi-phenotype prediction. Experimental results demonstrate that the F2TNet significantly improves the prediction of brain multi-phenotype and outperforms state-of-the-art methods. The code is available at https://github.com/CUHK-AIM-Group/F2TNet.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0615_paper.pdf

SharedIt Link: https://rdcu.be/dV58n

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72120-5_25

Supplementary Material: N/A

Link to the Code Repository

https://github.com/CUHK-AIM-Group/F2TNet

Link to the Dataset(s)

N/A

BibTex

@InProceedings{He_F2TNet_MICCAI2024,
        author = { He, Zhibin and Li, Wuyang and Jiang, Yu and Peng, Zhihao and Wang, Pengyu and Li, Xiang and Liu, Tianming and Han, Junwei and Zhang, Tuo and Yuan, Yixuan},
        title = { { F2TNet: FMRI to T1w MRI Knowledge Transfer Network for Brain Multi-phenotype Prediction } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15011},
        month = {October},
        page = {265 -- 275}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The authors aim to use the mre cost efficient T1-w modality to predict phenotypes in the same quality as the more ‘expensive’ fMRI. To do so, the authors propose a model based on knowledge transfer to obtain best of both worlds.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is clear. The structure is clear. The approach is quite innovative and well thought for using two modalities which cannot be translated one from the other.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Problem formulation is not well jsutified. Table 1 is not well structured and comparison style is weak. Please see comments below for further details.

  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    No code provided. limited ability to reproduce.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    While the paper is good in general. There are many limitations and it seems that is more suited for a MICCAI workshop than the main conference.

    1 - In my opinion, the model is over-complicated. If the authors plan to bring best of both worlds why would they simply train a model to translate the features from the transformer for T1-w MRI into theoutput features of the transformer from fMRI. If one knows that the features from fMRI yield very good prediction accuracy then one would only work on the high dimensional feature space. Then the problem become a very simple knowledge transfer problem. 2- Table1: I do not understand why the authors need to report results from the inference using fMRI since the entire paper premise is to use the cheaper T1-w MRI for inference. Also 0.29 (as an example) is not a very good PCC value. The authors need to argue why these values are good enough. 3- related to the last remark: the premise of the paper is to use cheaper modality to predict phenotype in with an accuracy closer to the scenario where the more expensive modality is used. However, the authors do not clearly quantify there results? how can we know that the obtained results are cost efficient enough? It would be good to report the ratio cost-to-efficency if one bases the paper on that. 4- Table1 : i still do not agree with the comparison style. I do not think the methodology of comparison makes sense. Shouldn’t the authors report PCC and RMSE on models trained with fMRI and T1-w and tested with T1-w ? I think it is trivial that a model would score lower if trained and tested both on T1-w. The authors propose a multimode framework, therefore they should compare against a multi-modal framework. While the authors might argue that a multimodal technique is non existant, I believe adapting the existing methods to concatenate both modalities would be at least an attempt to compare in a more fair way. 5- The authors completely rule out any discussion of the results. That would make the paper hard to fully understand. Even thought the number of pages is limited in such conference, a discussion is too important to neglect. 6-The authors do not provide a repository for code reproducibility. Minor: Fig1 legend is not very clear and coloring the arrows would be better way to make it more understandable.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Paper is not really suited for the main conference. It is more suited for a workshop. Several things are missing or unjustified. Please see my comments below.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The authors made significant changes to clarify their manuscript



Review #2

  • Please describe the contribution of the paper

    The authors propose a knowledge transfer network (F2TNet) from FMRI to T1w MRI that aims to address an important problem in systems neuroscience: how to accurately analyze brain multi-phenotypes in a cost-effective and efficient manner. The network comprises two modules: the Phenotypes-guided Knowledge Transfer (PgKT) module and the Modality-aware Multi-phenotype Prediction (MpP) module. The PgKT module achieves adaptive FMRI to T1w MRI by solving the bipartite graph-matching problem, thus enabling adaptive PgKT achieves adaptive knowledge transfer from FMRI to T1w MRI by solving the bipartite graph-matching problem, thus enabling effective multi-phenotype prediction. MpP, in turn, enriches the phenotype coding information and decomposes the coding to achieve accurate multi-phenotype prediction.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The article’s strengths are as follows: 1.The authors propose a knowledge transfer network, F2TNet, to address an important problem in systems neuroscience. This network is designed to migrate knowledge from FMRI images into T1wMRI. The F2TNet is intended to facilitate the accurate analysis of multiple phenotypes of the brain in a cost-effective and efficient manner. This represents an interesting attempt in the field. F2TNet combines two modules: the Phenotypes-guided Knowledge Transfer (PgKT) module and the Modality-aware Multi-phenotype Prediction (MpP) module. The PgKT module achieves this goal by solving the bipartite graph matching problem. This enables adaptive knowledge transfer from FMRI to T1w MRI. This innovation enables more efficient knowledge transfer from FMRI to T1w MRI and provides the basis for multi-phenotype prediction. The MpP module enriches the phenotypic coding information and decomposes the coding for accurate multi-phenotype prediction. The innovations in this module facilitate an improvement in the accuracy of prediction of multiple phenotypes, leading to a better understanding of multiple features of the brain. 2.As evidenced by experimental studies, F2TNet not only markedly enhances the accuracy of brain multi-phenotype prediction but also outperforms existing state-of-the-art methods while markedly reducing the prediction cost.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The problems with this paper are as follows: Figure 2 raises several issues that require further clarification. Firstly, the (a)PgKT and (b)MpP modules are not labelled. It is recommended that the sections belonging to (a) and (b) be marked with boxes. Secondly, the two occurrences of F_s^’ and F_t^’ in the figure appear are equal? Thirdly, the first half of Figure 2 is not adequately explained, particularly in relation to the input section and the Linear Projection. Chapter2: In 2 chapters, the letters used for the data are too similar resulting in some reading difficulties. In section 2.1 Cross-modality Alignment, “Gbm consists of an Affinity layer … a Sinkhorn layer” does not illustrate the layers involved. In chapter 2.1 Modality Knowledge Transfer, the dimensions of H_s, H_t are wrong, according to the author’s description, the dimensions of H_s, H_t should be K×N instead of K×n. In chapter 2.2, Modality-aware Phenotype Codes Decomposition, there is an inconsistency in the name of the multi-head attention mechanism, which is referred to as MAttn followed by MHAttn, and the second MHAttn (,,*) is not followed by punctuation. In chapter 2.2 Multi-phenotype Prediction, “multi multi-layer perceptron (MLP)” is written with an extra multi. Equation 4 lacks punctuation. In chapter 2.3 there is a grammatical error in “a hyperparameter α and β”. The formula 5 subscripts in italics are not harmonized with the format of the table below the other formulas. Chapter3: In chapter 3.1 Preprocessing, Angaggr_Unadj is inconsistent with the case in Figure 3. In 3.2 Comparison Experiments, the unclear presentation of the experiments and settings includes: i. The authors chose six phenotypes for the experiments without explaining why; ii. “In addition, we sequentially … experiments,” the presentation is unclear as to whether it is a case of two phenotypes out of twelve to predict four or two out of six to predict four. Third, “Subsequently, using phenotypes #7-#12 … method,” it is unclear whether all six phenotypes are the remaining six? In 3.2 Ablation experiments the experimental setup is not perfect, e.g. ablation experiments for phenotype activate map and ablation experiments for residual loss function should be added.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The suggestions are as follows:

    1. Please refine Figure 2 by labeling parts (a) and (b) in the figure
    2. Please pay attention to minor errors and standardization issues in writing such as multi multi-layer perceptron (MLP).
    3. Please improve your ablation experiments, and suggest adding ablation experiments for components in the PgKT module and ablation experiments for different loss functions.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This is indeed an intriguing endeavor in the field, showcasing the model’s robust capability in effectively addressing real-world challenges. However, there are several minor issues with the paper, including an incomplete Figure 2, unclear explanations, and unfinished experiments, resulting in a less favorable evaluation from my end.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The authors propose a method that performs multi-phenotype prediction by utilizing fMRI and T1w data at training time but only T1w data at inference time. The paper includes extensive quantitative evaluation against competing methods and also qualitative visualizations.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1) The task that this paper is trying to solve (i.e. knowledge transfer from fMRI to T1w driven by a downstream task) is novel and interesting. 2) The proposed method is well described and neatly presented. 3) The quantitative and qualitative evaluations affirm the performance of the proposed approach.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    There are some issues with the clarity of the reported experimental results. Please see further details in the “comments for the authors” section.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    The authors claim in the rebuttal that they will publicly release their code upon acceptance.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    • In Figure 2, it is unclear which part of the figure corresponds to the knowledge transfer part of the pipeline and which part corresponds to the prediction part.
    • In Table, the authors state the best results are highlighted. Only, the results of the proposed method are highlighted (row 13) but there are metrics for which the proposed method has not achieved the best results. Additionally, the authors state “F2TNet outperforms even when only fMRI is utilized (9th row).”, which also appears to be incorrect as the fMRI-only approach appears to yield better results for the majority of the metrics. Intuitively, using the same data for training and testing should provide a threshold for the knowledge-transfer approach. I would like some clarifications and further discussion from the authors here.
    • What are the implications of the ability to transfer knowledge from fMRI (a functional modality) to T1w (a structural modality)? As the authors mention leveraging T1w solely does not yield good performance as the phenotypes that are being predicted are potentially associated better with function rather than structure. It is a bit counter-intuitive and not straightforward to understand what is the type of knowledge that is being transferred. Additionally, how would that be affected if a similar framework was deployed on a different task (e.g. something that is more relevant to structure) or if it was done in a manner that was not driven by a particular downstream task? I would appreciate it if the authors could discuss these issues.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The novelty of the task and the formulation of the method.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The authors have mostly addressed the reviewers’ comments. Regarding my comments about Table 1 (also made by R4), they authors claim that they have revised the table to clarify certain aspects. Naturally, without seeing the updated table it is difficult to confirm whether the changes are successful but the authors are not at fault for that. Regarding my comments on discussing the general implications of the paper (in combination with R4’s comments on the general lack of discussion), I still feel that this is an important aspect of any paper and it should be included in the manuscript. I understand that space limitations exist but at least a certain level of discussion should be present. Overall, I am inclined to keep my initial score (weak accept).



Author Feedback

We thank the reviewers for their comments.

R4(Q2,Q4)&5(Q2): Clarify the Table 1 results by multimodal training, T1w testing, and why the results of testing with fMRI are shown. Sorry for the confusion caused by Table 1. We have reorganized it to align results with consistent training/testing modalities. 1) fMRI for Prediction: Previous studies used fMRI and achieved satisfactory results (rows 1,3,5,7). For comparison, we show our fMRI-only results (row 9) and obtain the best performance, highlighting the effectiveness of our proposed phenotypes activation maps. 2) T1w for Prediction: Due to the lower cost of Tw, it is a more practical setting to use T1w for prediction, which still shows significant challenges in existing works (rows 2,4,6,8). 3) Knowledge Transfer: Hence, we transfer knowledge from high-cost fMRI to low-cost T1w (training: fMRI+T1w) and achieve the best performance using T1w for testing (row 13) (outperforming existing works only using T1w (rows 2,4,6,8)), fulfilling our original intention. 4) We aim to address missing high-cost fMRI rather than multimodal fusion, so we didn’t present fusion results. 5) Phenotype prediction is a recognized challenging task [3,15,20]. Previous fMRI-based predictions typically have PCC<0.2 and T1w-based predictions are even lower. Thus, our results are good enough.

R4(Q5)&5(Q3): Discussion about the implication of the ability to transfer knowledge from fMRI to T1w. Brain structure determines function, and function determines phenotype. The unclear nonlinear relation between structure and function is a key focus in neuroscience. Through the downstream task (phenotype prediction), we transferred the knowledge from fMRI to T1w, not only exploring this relation but also establishing the mapping from structure to phenotype for precise prediction. Our method can be applied to other tasks, e.g., brain mental disorders classification.

R1&4&5(Q1): Clarify the Fig.2. We will label the PgKT and MpP modules with boxes in Fig. 2. The PgKT includes cross-modality alignment, phenotypes activation maps, and modality knowledge transfer. The MpP involves modality-aware phenotype codes decomposition, multi-phenotype prediction, and modality-aware phenotype.

R1: Writing errors, letters explanation. We will fix all the writing errors. Fs’ and Ft’ are equal twice in Fig.2. F/T represent image/phenotype features. The letter superscript {‘} indicates features after phenotypes activation maps bootstrapping and {s/t} denote fMRI/T1w features.

R1: Clarify the Linear Projection and Sinkhorn layer. The Linear Projection is an MLP layer to project image features into the transformer space. The Sinkhorn layer, commonly used in graph matching [23], facilitates intermodal node matching.

R1: Phenotypes’ selection. Based on [15], we randomly selected 12 phenotypes predictable through fMRI. To justify that our method is robust to the number of phenotypes, we eliminated 2 phenotypes and predicted 4 at the same time. Phenotypes #7-#12 are the remaining 6 phenotypes.

R1: Ablation studies. Compared to our full method (PCC=0.24), removing phenotype activation maps and the modality knowledge transfer loss (Lmc) reduces performance (PCC=0.22,0.08), verifying each component’s effectiveness.

R4(Q1): Explain why simple modality transfer was not conducted and why our method is not overcomplicated. Predicting phenotypes is not a simple task [3,15,20]. Simple modality transfer (PCC=0.09) performs much lower than our method, proving it can’t achieve precise results. Our ablation studies show that each module is useful, validating that our method is not overcomplicated and phenotype prediction is indeed a complex task.

R4(Q3): Cost-to-efficiency. Inference times are as follows {[17,7], our}: {0.47s, 0.53s, 0.32s}. Our method achieves shorter inference times, higher accuracy, and lower data acquisition costs, enhancing its clinical value.

R4(Q6): Code reproducibility. Our code will be made public upon acceptance.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The rebuttal has addressed the concerns by the reviewers, and all reviewers have ranked this paper as “weakly accept”, which I also find no issues.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The rebuttal has addressed the concerns by the reviewers, and all reviewers have ranked this paper as “weakly accept”, which I also find no issues.



back to top