List of Papers Browse by Subject Areas Author List
Abstract
Building a generalist model for neuron instance segmentation from electron microscopy (EM) volumes holds great potential to accelerate data processing and analysis in connectomics. However, the diversity in visual appearances and voxel resolutions present obstacles to model development. Meanwhile, prompt-based foundation models for segmentation struggle to achieve satisfactory performance due to the inherent complexity and volumetric continuity of neuronal structures. To address this, this paper introduces SegNeuron, a generalist model for dense neuron instance segmentation with strong zero-shot generalizability. To this end, we first construct a multi-resolution, multi-modality, and multi-species volume EM database, named EMNeuron, consisting of over 22 billion voxels, with over 3 billion densely labeled. On this basis, we devise a novel workflow to build the model with customized strategies, including pretraining via multi-scale Gaussian mask reconstruction, domain-mixing finetuning, and foreground-restricted instance segmentation. Experimental results on unseen datasets indicate that SegNeuron not only significantly surpasses existing generalist models, but also achieves competitive or even superior results with specialist models. Datasets, codes, and models are available at https://github.com/yanchaoz/SegNeuron.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0518_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0518_supp.pdf
Link to the Code Repository
https://github.com/yanchaoz/SegNeuron
Link to the Dataset(s)
https://github.com/yanchaoz/SegNeuron
BibTex
@InProceedings{Zha_SegNeuron_MICCAI2024,
author = { Zhang, Yanchao and Guo, Jinyue and Zhai, Hao and Liu, Jing and Han, Hua},
title = { { SegNeuron: 3D Neuron Instance Segmentation in Any EM Volume with a Generalist Model } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15008},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper introduces SegNeuron, a generalist model for neuron instance segmentation in electron microscopy (EM) volumes that achieves notable improvement over existing models. The key contributions are the development of a multi-resolution, multi-modality, and multi-species volumetric EM database called EMNeuron. This large-scale dataset underpins the model’s robust training and generalization capabilities. Additionally, the paper proposes innovative training strategies, such as multi-scale Gaussian mask reconstruction for pretraining and domain-mixing finetuning, enhancing the model’s performance across diverse data distributions and spatial resolutions. These methodologies allow SegNeuron to substantially outperform existing generalist models.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
A. The architecture of SegNeuron and its training methodologies are particularly noteworthy. The model uses a multi-scale Gaussian mask reconstruction during its pretraining phase, which is a novel approach allowing it to learn robust representations without distorting the distribution of the data.
B. The paper details innovative data utilization strategies such as domain-mixing finetuning and frequency and spatial domain mixing. These techniques are designed to enhance the model’s ability to generalize across different visual appearances and voxel resolutions. Such strategies are crucial for dealing with the variability in EM data due to different species, tissue types, and preparation methods.
C. The construction of the EMNeuron dataset is a major strength of this paper. This dataset is not only large-scale, containing over 22 billion voxels with more than 3 billion densely labeled, but also diverse, including multi-resolution, multi-modality, and multi-species EM volumes. This diversity is crucial for developing a generalist model like SegNeuron, as it ensures robustness and generalizability across different data distributions and spatial resolutions.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
A. The paper describes a method where the low-frequency amplitude component is substituted with that from a sampled auxiliary input, which is intended to maintain discriminative boundary information and enhance the style and texture of the training data. However, this substitution may potentially generate unrealistic training samples, resulting in the trained model not being suitable for realistic data. Additionally, the manuscript lacks a detailed explanation of the determination of the frequency mixing ratio. And lack the comparison of model performance under different such ratios.
B. The manuscript employs a strategy of masking data with randomly sized patches, which could potentially allow the model to interpolate and recover image areas from adjacent unmasked pixels, particularly given the redundant information in images. If the patches are too small, there is a concern that the model may primarily learn to interpolate these areas rather than extracting richer, more generalized features from a broader context. It would be beneficial for the authors to discuss this.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The manuscript presents an innovative approach to enhancing style and texture in training data through the substitution of low-frequency amplitude components with those from a sampled auxiliary input. While this method aims to preserve discriminative boundary information, there is a potential risk of generating unrealistic training samples that may not adequately represent real-world scenarios. This could significantly impact the model’s applicability and effectiveness in practical settings. Moreover, the manuscript does not provide a clear methodology for determining the frequency mixing ratio, nor does it evaluate the model’s performance across different ratios. To strengthen the submission, it is recommended that the authors include a detailed justification of the chosen frequency mixing ratio and conduct experiments to assess the impact of varying these ratios on model performance. This would provide a more comprehensive understanding of the method’s robustness and its potential limitations in clinical and other real-world applications.
Additionally, the use of randomly sized patches for data masking raises concerns about the model’s ability to generalize beyond simple interpolation of local image features, particularly in contexts with high data redundancy. The possibility that the model might predominantly learn interpolation rather than feature extraction could undermine the depth and utility of the learned representations. It would be valuable for the authors to explore and discuss the effects of different patch sizes and masking strategies on the model’s ability to extract meaningful and generalized features. A comparative analysis, possibly supported by visual or quantitative results, could elucidate the trade-offs involved and help optimize the masking strategy for enhanced model training and performance.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The recommendation is primarily influenced by the concerns about unrealistic training samples and the model’s potential to learn interpolation. A detailed rebuttal addressing the generation of training data realism and the justification for parameter choices could substantiate the paper’s contributions and strengthen its candidacy for acceptance.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
This work presents a Neuron Instance Segmentation framework, SegNeuron. It constructs a multi-resolution, multi-modality, and multispecies volume EM database, named EMNeuron, consisting of over 22 billion voxels, with over 3 billion densely labeled. They also design a new framework for neuron instance segmentation.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
They construct a large database with 16 EM datasets and 22B+ voxels, and customized a pipe flow including pretraining via multi-scale Gaussian mask reconstruction, domain-mixing finetuning, and foreground-restricted instance segmentation.The results demonstrate both efficiency and accuracy of the model.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The framework only considers neuron segmentation. However, as a generalist model for EM segmentation, there are some other EM property needs to be considered. For example, the organnelle segmentation.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The model should include the organnelle segmentation tasks to improve its generalizability.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
They construct a large dataset, and develop an efficient and effective framework for neuron segmentation. Its a well-organized work.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
The manuscript titled, “SegNeuron: 3D Neuron Instance Segmentation in Any EM Volume with a Generalist Model” demonstrates a methodology that enables the segmentation of neurons using volumetric EM datasets with good zero-shot generalization capabilities.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengths of the paper lie in their novel workflow in building the model. Specifically, the manuscript discusses their customization in designing (1) a Gaussian noise addition-recovery proxy task for model pretraining, which builds mask reconstruction in a multi-scale manner without distribution distortion, (2) frequency and spatial domain mixing finetuning to generate training data, (3) foreground-restricted instance segmentation to remove noisy values. Furthermore, the performance of the model is excellent in segmenting dense neurons, which are typically difficult to segment due to their unconventional shape as well as their volumetric continutity in 3D datasets. The biomedical scope of this work is high based on the application of this model to volumetric neuronal studies which begin with segmentation. Additionally, the manuscript also demonstrates the superior performance and good generalizability of the model for various datasets.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The weakness of the paper lies in the extremely small figure panels showcasing the segmentation quality. It would have been better to see larger panels to emphasized differences comparing the different generalist models.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The manuscript titled, “SegNeuron: 3D Neuron Instance Segmentation in Any EM Volume with a Generalist Model” demonstrates a methodology that enables segmentation of neurons using volumetric EM datasets with good zero-shot generalization capabilities. The main strengths of the paper lie in their novel workflow in building the model. Specifically, the manuscript discusses their customization in designing (1) a Gaussian noise addition-recovery proxy task for model pretraining, which builds mask reconstruction in a multi-scale manner without distribution distortion, (2) frequency and spatial domain mixing finetuning to generate training data, (3) foreground-restricted instance segmentation to remove noisy values. Furthermore, the performance of the model is excellent in segmenting dense neurons, which are typically difficult to segment due to their unconventional shape as well as their volumetric continutity in 3D datasets. The biomedical scope of this work is high based on the application of this model to volumetric neuronal studies which begin with segmentation. Additionally, the manuscript also demonstrates the superior performance and good generalizability of the model for various datasets. The weakness of the paper lies in the extremely small figure panels showcasing the segmentation quality. It would have been better to see larger panels to emphasize differences comparing the different generalist models. I am satisfied with the condition the manuscript is currently in and recommend it for publication.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Novelty and biomedical relevance is high.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
N/A
Meta-Review
Meta-review not available, early accepted paper.