List of Papers Browse by Subject Areas Author List
Abstract
Understanding the transport of molecules in the brain \emph{in vivo} is the key to learning how the brain regulates its metabolism, how brain pathologies develop, and how most of the developed brain-targeted drugs fail. Two-photon microscopy – the main tool for \emph{in vivo} brain imaging – achieves sub-micrometer resolution and high image contrast when imaging cells, blood vessels, and other microscopic structures. However, images of small and fast-moving objects, e.g. nanoparticles, are ill-suited for analysis of transport with standard methods, e.g. super-localization, because of (i) low photon budgets resulting in noisy images; (ii) severe motion blur due to slow pixel-by-pixel image acquisition by t-photon microscopy; and (iii) high density of tracked objects, preventing their individual localization.
Here, we developed a deep learning-based estimator of diffusion coefficients of nanoparticles directly from movies recorded with two-photon microscopy \emph{in vivo}.
We’ve benchmarked the method with synthetic data, model experimental data (nanoparticles in water), and \emph{in vivo} data (nanoparticles in the brain).
Deep Learning robustly estimates the diffusion coefficient of nanoparticles from movies with severe motion blur and movies with high nanoparticle densities, where, in contrast to the classic algorithms, the deep learning estimator’s accuracy improves with increasing density.
As a result, the deep learning estimator facilitates the estimation of diffusion coefficients of nanoparticles in the brain \emph{in vivo}, where the existing estimators fail.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2899_paper.pdf
SharedIt Link: https://rdcu.be/dV1MD
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72069-7_20
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/2899_supp.zip
Link to the Code Repository
https://github.com/kirkegaardlab/2photodiffusion
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Kir_In_MICCAI2024,
author = { Kirkegaard, Julius B. and Kutuzov, Nikolay P. and Netterstrøm, Rasmus and Darkner, Sune and Lauritzen, Martin and Lauze, François},
title = { { In vivo deep learning estimation of diffusion coefficients of nanoparticles } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15002},
month = {October},
page = {206 -- 215}
}
Reviews
Review #1
- Please describe the contribution of the paper
In this paper, the authors proposed a deep learning (DL) based estimator for estimating diffusion coefficients (DC) of brain-targeted nanoparticles (NP). The DL model takes a sequence of gray-scale images as input and estimate the DCs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The authors have presented a novel application.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The paper is not well-presented. The proposed method is not novel. The results are not convincing.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The implementation details of the proposed method are missing. The data is not publicly available.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- The authors may want to provide a figure to show the DL model.
- More recent methods should be compared, i.e., the only method used for comparison has been published in 1996.
- The authors should provide implementation detail. For example, the framework (tensorflow or pytorch) used to implement the DL model. The hyper parameters (batch size, learning rate and epochs) should be provide.
- For the reproducibility, if the authors could make their code publicly available, it would be great.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Strong Reject — must be rejected due to major flaws (1)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is not well-organized. The writing is poor. The technical part is not solid. The implementation is not detailed. The results are not convincing.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper proposes a deep learning method for estimating the diffusion coefficients of nanoparticles in the brain from two-proton microscopy. Standard particle tracking fails in this images due to high motion blur and high particle densities that are present in the in-vivo two-proton microscopy videos.
The method is validated using synthetic data, model experimental data (nanoparticles in water), and in vivo data (nanoparticles in the brain).
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The idea of using deep learning for estimating diffusive behaviour (ref from page2 second paragraph) but the use of deep learning in this particular setting appears new. The paper uses synthetic data with high and low diffusion coefficients for training. Testing is done on both synthetic data, experimental data (diffusion in water that can be controlled and estimated with classic methods) and in-vivo data (from the mouse brain). For the in-vivo experiments the results are consistent with existing literature (ref 4).
The paper has good practical application in preclinical studies of brain-targeted drugs.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The network architecture description - Fig 2 is not entirely clear explained.
- Is the autoencoder (i) trained separately on image data first ?
- Is the captive pooling part of the resent ?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The network architecture could be described in more details for being able to be reproduced (see comment above). The system data is well explained. The real data is acquired in a laboratory setting so it is more difficult to generate.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
I would recommend clarifying the network architecture and training. Also would be interesting to get some insights into the generality of the synthetic training data. For an outsider to the field, how rgeneral is this data for other experiments with two-photon microscopy images ?
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
While not an expert in this area, I found the paper well written, the idea relatively novel and the experiments convincing. Even though experiments are limited and somewhat narrow (particular type of data constant with their real data) it is a method that could be useful in practice and could easily be extended to other types of two-proton microscopy images.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
In this paper, the authors propose an innovative solution to estimate diffusion coefficients of nanoparticles in the in vivo brain using two-photon microscopy. They do this to tackle on the long term the estimation of diffusion coefficients in the brain. Technically, they train a Deep learning network to estimate diffusion coefficient from synthetic data movies of nanoparticles in water. They validate the results on synthetic data, on movies of nanobeads in waters and they test the trained network on movies of two-photon images of nanobeads in in vivo mouse brains before and after heart stop.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Using sound synthetic data and comparing with sound baselines, they propose a method to estimate diffusion coefficients from movies of in vivo brain mice that, contrary to classical particle tracking methods, improves with increasing particle density
- The authors control fully the way they train and validate their Deep learning algorithm: they train a Deep learning algorithm on controlled synthetic data of nanobeads in water with different diffusion coefficients, validate them on a similar setting, on real movies of nanobeads in water and on two-photo images of brain mice containing nanobeads before and after heart stop.
- The proposed solution outperforms classical algorithms and improves with increasing nanobead densities, contrary to classical tracking algorithms, rendering it more useable in a preclinical settings
- They tackle a fundamental problem, measuring diffusion coefficient in the brain, that will have strong repercussions in preclinical experiments
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The main weakness is the level of information given on the hyperpamareters of the Deep learning framework, and how these hyperparameters were chosen
- The authors don’t give enough details on the zero-knowledge baseline and about the corresponding a prioris
- The authors don’t give enough details on how is computed sigma, the error of the estimate, which is the second output of the network
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
For the Deep learning part, the authors have not provided any information about hyperparameters and they have not provided any source code. I think that it is possible for the authors to provide the list of hyperparameters used and the source code at least of the Deep learning model in python code. It would be very valuable.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
- On my opinion, the authors should add the hyperpameters used, at minimum the learning rate, the non-linearities. It would be nice for any reader to have, for example in Appendix, the structure of the network used. If possible, the authors should also give at least the Pytorch-like code of the neural network
- On my opinion, it is necessary to give more information on how is computed the error of the estimate sigma, which is the second output of the final MLP
- It would be great to have also a more through explanation on the zero-knowledge baseline (It may be obvious but I can’t figure it out for sure)
Minor:
- The authors should provide more clearly the size of the quantum dots and the used beads: from the reference (QTracker655), I found (from a quick search and I may be wrong) that it was a 20 nm size, but from figure 3, you speak of 50-nm nanobeads in water. Are they the same beads?
- In Table 1, the authors should give “zero-knowledge” instead of baseline, or at least give in the caption what they intend by “Baseline”
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I believe that this paper is outstanding and very complete, in the sense that it goes from physics simulation up to the Deep learning estimation of diffusion coefficient in in vivo two-photon microscopy images. Experiments are sound and look very novel and intriguing for a conference like MICCAI. I will be ready to update to a strong accept if the authors make the effort to provide more complete information on the Deep learning part of their work.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Strong Accept — must be accepted due to excellence (6)
- [Post rebuttal] Please justify your decision
I always think that the paper is of strong quality and original and relevant to the MICCAI community. The authors addressed the issue of the lack of information about the Deep learning part. They claimed that they will publish the code upon publication: I can only trust the authors here. I think the added information is sufficient and have updated the score accordingly
Author Feedback
We thank the reviewers for their critical assessment of our work. In the following we address their concerns.
Reproducibility
The main concern of all three reviewers is the lack of information given on the network structure and hyperparameters. In the paper, we do indeed only state the high-level elements of our approach and provide limited details. We wanted to focus on the novel approach to predicting diffusion coefficients rather than on the precise details of the networks we used. Nonetheless, we fully agree with the reviewers that these details should not be omitted. All details can be found in the source code which we will release upon acceptance of the paper.
The encoder part of our network consists of five standard convolutional layers with GeLU activations. This network is trained separately by minimizing an autoencoder L2 loss using an Adam optimizer with a learning rate of 5e-4 and a batch size of 512.
The predictor network uses a 1D ResNet with seven ResNet blocks and an initial kernel size of seven-time steps. The ResNet output is average-pooled and linearly transformed. The final layer outputs two numbers, which are mapped to positive by f(x) = 1 + ELU(x). The first of these numbers is the diffusion coefficient D and the second is the estimated error sigma. The loss [Eq. (2) in main text] ensures that sigma is the optimal Gaussian noise standard deviation. This network is trained using an Adam optimizer with a learning rate of 0.01 and a batch size of 50. As we use synthetic data, we generate data on-the-fly and never show the network the same data twice.
Baselines
The reviewers expressed concern on how we defined the zero-knowledge baselines. Our synthetic datasets consist of videos with known diffusion coefficients sampled from a uniform range [D_low; D_high]. Our baselines are simply the numbers that result from always predicting the mean (D_low + D_high) / 2. In the main text, we give the example L1= <|x-0.75|> = 0.375 for the dataset with D sampled from [0;1.5], i.e. L1 = 1/1.5 * \int_0^1.5 |x - 0.75| dx.
Error computation
As mentioned above, the error estimate sigma is just one of two outputs (D, sigma) of our prediction network, both of which are forced positive by a non-linearity. This output is trained to be an accurate predictor of the error as needed to minimize the Gaussian likelihood loss (Eq. 2 of main text).
Sizes of nanoparticles
Reviewer 3 asks about the sizes of the quantum dots and fluorescent nanobeads. The quantum dots and nanobeads have 25 nm and 50 nm diameters, respectively.
Generality of simulations
Reviewer 4 asks to the generality of our synthetic simulations. The main limitation is our focus on microscopic videos in which there are no other objects visible. This restriction can possibly be relaxed by training on masked videos. Furthermore, we have tuned the intensity of the QDs, the scanning speed, etc. to that of our experimental setup, but such parameters are simple to change in our approach.
Comparisons
Reviewer 5 asks if we can compare to more previous literature. To our knowledge, the determination of diffusion coefficients in previous literature on nanobeads is always done using particle tracking (which we do compare to). Our work is fundamentally different from this approach, and as we show, does not only improve the accuracy but also demonstrates a completely different scaling with density of particles.
Implementation details
We use PyTorch for both training, model specification, and simulations. All is run, end-to-end, on GPUs. As already mentioned, we will release our source code upon acceptance.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The authors present a novel deep learning (DL) based estimator for determining the diffusion coefficients (DC) of brain-targeted nanoparticles (NPs) using gray-scale image sequences. This approach is particularly notable for its application to in vivo two-photon microscopy (2PM) data, overcoming limitations associated with high motion blur and NP density that hinder traditional methods.
The proposed DL-based estimator is a significant advancement in estimating diffusion coefficients from high-motion blur and high-density nanoparticle images. This represents a new angle for the MICCAI community.
The method was rigorously evaluated against synthetic data, model experimental data, and in vivo data, demonstrating robustness and superior performance compared to traditional methods.
The technique is directly applicable to in vivo studies addressing a critical challenge in neurodegenerative disease research and drug delivery studies.
The authors have addressed reproducibility concerns by providing detailed methodological information and committing to open-source code release.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The authors present a novel deep learning (DL) based estimator for determining the diffusion coefficients (DC) of brain-targeted nanoparticles (NPs) using gray-scale image sequences. This approach is particularly notable for its application to in vivo two-photon microscopy (2PM) data, overcoming limitations associated with high motion blur and NP density that hinder traditional methods.
The proposed DL-based estimator is a significant advancement in estimating diffusion coefficients from high-motion blur and high-density nanoparticle images. This represents a new angle for the MICCAI community.
The method was rigorously evaluated against synthetic data, model experimental data, and in vivo data, demonstrating robustness and superior performance compared to traditional methods.
The technique is directly applicable to in vivo studies addressing a critical challenge in neurodegenerative disease research and drug delivery studies.
The authors have addressed reproducibility concerns by providing detailed methodological information and committing to open-source code release.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
Two reviewers were very enthusiastic about the paper and I agree that this paper brings a new angle to MICCAI which can be be appreciated. Accept.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
Two reviewers were very enthusiastic about the paper and I agree that this paper brings a new angle to MICCAI which can be be appreciated. Accept.