List of Papers Browse by Subject Areas Author List
Abstract
The fundamental question in neuroscience is to understand the working mechanism of how anatomical structure supports brain function and how remarkable functional fluctuations emerge ubiquitous behaviors. We formulate this inverse problem in the realm of system identification, where we use a geometric scattering transform (GST) to model the structure-function coupling and a neural Koopman operator to uncover dynamic mechanism of the underlying complex system. First, GST is used to construct a collection of measurements by projecting the proxy signal of brain activity into a neural manifold constrained by the geometry of wiring patterns in the brain. Then, we seek to find a Koopman operator to elucidate the complex relationship between partial observations and behavior outcomes with a relatively simpler linear mapping, which allows us to understand functional dynamics in the clich'e of control system. Furthermore, we integrate GST and Koopman operator into an end-to-end deep neural network, yielding an explainable model for brain dynamics with a mathematical guarantee. Through rigorous experiments conducted on the Human Connectome Project-Aging (HCP-A) dataset, our method demonstrates state-of-the-art performance in cognitive task classification, surpassing existing benchmarks. More importantly, our method shows great potential in uncovering novel insights of brain dynamics using machine learning approach.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2162_paper.pdf
SharedIt Link: https://rdcu.be/dV1PU
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_48
Supplementary Material: N/A
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Cho_Understanding_MICCAI2024,
author = { Chow, Chiyuen and Dan, Tingting and Styner, Martin and Wu, Guorong},
title = { { Understanding Brain Dynamics Through Neural Koopman Operator with Structure-Function Coupling } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15002},
month = {October},
page = {509 -- 518}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper introduces a method named Scattering Neural Koopman Operator. It leverages Koopman operator theory and geometric scattering transform to couple structural brain network and brain activity signal to model the global dynamics of the brain system. The combination of these methods provides a comprehensive understanding of complex brain functions and dynamics.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The integration of geometric scattering transforms with the Koopman operator framework is novel in the neuroscience field, providing a new way to study brain dynamics that combines structural and functional data effectively.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1)The complexity of the methods used may make it difficult for those not familiar with both deep learning and dynamical systems theory to fully understand or implement the approach. 2)While the model performs well on the HCP-A dataset, the paper does not discuss its applicability to other types of datasets or conditions outside of the specific tasks tested. 3) There is limited discussion regarding the potential limitations or drawbacks of the model, including how it handles the inherent nonlinearity of brain dynamics beyond the datasets used. 4) The comparison between different methodologies focuses exclusively on the outcomes of cognitive task classification. It does not directly address or compare how different approaches handle the coupling of structural brain networks with global brain dynamics. From my view, this aspect is important understanding the full implications and uniqueness of the proposed method, particularly how it models these dynamics compared to existing techniques.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
1) Please make comparisons with existing models or methods on diverse datasets to demonstrate the robustness and versatility of the proposed approach. 2) Provide a thorough discussion of the limitations and potential drawbacks of the model.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Reject — should be rejected, independent of rebuttal (2)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The motivation is good but the experiments are insufficient.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
The authors have addressed my concerns.
Review #2
- Please describe the contribution of the paper
The authors use a geometric scattering transform (GST) to model the structure-function coupling and a neural Koopman operator to uncover dynamic mechanism of the underlying complex system. The authors propose a new method, called Scattering Neural Koopman Operator (SKoop-C), to model the global dynamics of brain systems by combining structural connectivity and BOLD signals. The method was experimented on the Human Connectome Project-Aging (HCPA) dataset and achieved state-of-the-art performance in cognitive task classification.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
The authors combine Koopman operator theory and geometric scattering transforms to propose a new data-driven approach to model the coupling of structural and functional signals in the brain.
-
State-of-the-art performance is achieved on the HCP Age dataset, demonstrating the effectiveness of the method in cognitive task classification.
-
And an interpretable visualization of activated regions in different task states is provided by a control module.
-
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The generalization ability of the proposed model is not discussed in the paper. The authors are advised to test the model on different datasets (eg. HCP) to verify its generalization ability.
-
The paper does not discuss the computational complexity of the SKoop-C model, including training time and resource consumption. It is recommended that the authors provide this information.
-
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Please reference 6. Please list the main weaknesses of the paper.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This article is innovative enough that the experimental results exceed existing benchmarks and provide interpretive visualization of active brain regions through control modules, which is particularly important for research in the field of neuroscience.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Reject — could be rejected, dependent on rebuttal (3)
- [Post rebuttal] Please justify your decision
1) The authors show that the problem of generalizability is solved (experiments on the HCP dataset), but this is not convincing. 2) Also the authors do not show results on training time and resource consumption. 3) The authors show that the code is publicly available, but I did not find it.
Review #3
- Please describe the contribution of the paper
Propose a physics-inspired learning-based brain dynamic modeling under the theoretical framework of geometric scattering transform, Koopman operator, and linear state control.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This is an interesting paper with strong mathematical root and novel theoretical perspective for understanding the brain dynamics in terms of state transition. To select a proper measurement space, the authors propose to utilize geometric scattering transform based on structural connectivity. In that particular space, the brain dynamics can be modelled by learning a linear Markov process on Koopman operator, control signal, and control matrix. Using the data of HCP-A and a comparison with several advanced learning-based methods, the authors prove the advantage of their modelling schemes in brain state classification. The interpretability from the framework provides certain level of neuroscience understanding.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The readablity of the paper can be improved by further enhancing the concept introduction, methodological motivation, and implementation details.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
-
Fig. 1 is currently not informative. Alternatively, the authors are suggested to use Fig. 1 to illustrate the key idea/concept of modelling the brain dynamics. E.g. How did the authors formulate the dynamics? How the geometric scattering transform and Koopman operator helps the modelling? Where and how does the real data fit to the formulted model? It could helps the reader to better undestand the details in the methods.
-
The logic in Section 2.2 is not clear. Why constructing measurements based on a GST with structural connectivity is a “judiciously selection”?
-
How to choose the window for the control signal? Should the windowed signal be before the given time point? Is a instaneous window the best choice?
-
I’m curious about why the control signal can be estimated from brain signal itself rather than from an external process? Why the control process is Markov?
-
How was the data preprocessed? Did the authors filter the fMRI signal? What is “fMRI (4,846 time series)” when AAL-90 was used?
-
Other details about training, neural network implementation, the realization of GST is missing.
-
What is the unit for the frequency in Fig 3(b)?
-
There should be only 4 status in Fig. 6. BTW, is the acc, f1, etc a global averaged value from a 4-classification?
-
The meaning of Italic B should be explained.
-
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The thoereitcal and methodological novelity of this paper.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Accept — should be accepted, independent of rebuttal (5)
- [Post rebuttal] Please justify your decision
The thoereitcal and methodological novelity of this paper. This work could inspire the field and could be a strong paper to be presented in MICCAI.
Author Feedback
We appreciate all the constructive comments and suggestions. We are committed to incorporating all the valuable feedback into the final version of our paper. The code has been released in Anonymous Github and will be published after acceptance.
R1
The illustration of Fig. 1:
We appreciate your very constructive suggestion. We will improve Fig. 1 by incorporating all comments in the final version.
Why constructing measurements based on a GST with structural connectivity is a “judiciously selection”?
First, the use of GST aligns closely with our vision of the human brain, where we conceptualize that cognition and behavior emerge from spontaneous functional fluctuations supported by neural circuits of physically interconnected brain regions. Second, GST offers a flexible multi-scale window to elucidate the SC-FC coupling mechanism with great neuroscience inght and mathematical guarantee. In this regard, we emply GST to generat a set of parital observations from BOLD signals. We will make it clear in the final version.
The choice of the window for the control signal:
The window size for the control signal is chosen empirically to balance coputatinal cost and neuroscience domain knowledge. Note, we fixed this hyperparameter for all experiments.
The control is estimated from brain signal itself…
The control signal is jointly estimated from the brain signal as the part of our end-to-end deep model. We like this reviewer’s idea of using external predefined control force. Our model is flexible to incorporate external process. There is converging consensus that the control process is considered Markov in neuroscience field, as witnessed by an increasing number of computational work on Markov modeling for functional dynamics.
R3&R4
The generalization ability…
Thanks for the very constructive comments, we’re thrilled to share that our model has been tested on the HCP dataset for recognizing seven working-memory tasks. Our results consistently outperform other methods, matching the performance of BolT. We’re committed to including these HCP experiments in final version.
R4
The background of deep learning and dynamical systems…
We will add more detailed background of deep learning and dynamical systems to the final version.
Lacking discussion on potential limitation
We apologize that we have not suffiently discuss the limitation of our work, due to page limit. But we are glad to discuss the potential pitfall and solution in the rebuttal letter and include this in the final version.
Scalablity in estimating Koopman operator. It is computational expensive to estimate Koopman operator as more and more brain regions are taken into account. The possible silution is to introduce additional constraints to make it scale up to fine-grained atlas or even voxel-based manner.
Lack of comparison with analytic approaches in neuroscience field. We will benchmark the findings on SC-FC coupling mechanism in various experiment settings including disease connectome in clinical applications.
Prior studies, such as Ref. [15], have showcased the efficacy of the Koopman operator in approximating nonlinear dynamics within spatio-temporal systems. And, various studies, including the present work, have affirmed its effectiveness in modeling the nonlinear behavior of brain systems.
Lacking comparison with other SC-FC coupling methods
As a data-driven deep model, our primary focus is on benchmarking prediction accuracy against existing state-of-the-art methods. We fully concur with the reviewer’s suggestion that comparing the coupling mechanism between SC and FC with current analytic SC-FC coupling methods is an important aspect that we plan to address in future work.
R1&R3&R4 The details of our data process and training …
The code has been released in Anonymous Github. The frequency has no physical unit and It’s defined by the eigenvalues of graph Laplacian. We will add the detailed data processing tool in the final version.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The paper tackles the problem of modeling brain functional signals as a dynamic system. It utilizes geometric scattering to transform the fMRI signal, then uses Koopman operator framework to model the transformed signal as brain states. It has good potential to investigate brain functional-structural architecture in a control system perspective.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The paper tackles the problem of modeling brain functional signals as a dynamic system. It utilizes geometric scattering to transform the fMRI signal, then uses Koopman operator framework to model the transformed signal as brain states. It has good potential to investigate brain functional-structural architecture in a control system perspective.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A