List of Papers Browse by Subject Areas Author List
Abstract
The heterogeneity of neurological conditions, ranging from structural anomalies to functional impairments, presents a significant challenge in medical imaging analysis tasks. Moreover, the limited availability of well-annotated datasets constrains the development of robust analysis models. Against this backdrop, this study introduces a novel approach leveraging the inherent anatomical symmetrical features of the human brain to enhance the subsequent detection and segmentation analysis for brain diseases. A novel Symmetry-Aware Cross-Attention (SACA) module is proposed to encode symmetrical features of left and right hemispheres, and a proxy task to detect symmetrical features as the Symmetry-Aware Head (SAH) is proposed, which guides the pretraining of the whole network on a vast 3D brain imaging dataset comprising both healthy and diseased brain images across various MRI and CT. Through meticulous experimentation on downstream tasks, including both classification and segmentation for brain diseases, our model demonstrates superior performance over state-of-the-art methodologies, particularly highlighting the significance of symmetry-aware learning. Our findings advocate for the effectiveness of incorporating symmetry awareness into pretraining and set a new benchmark for medical imaging analysis, promising significant strides toward accurate and efficient diagnostic processes. Code is available at https://github.com/bitMyron/sa-swin.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/2579_paper.pdf
SharedIt Link: pending
SpringerLink (DOI): pending
Supplementary Material: https://papers.miccai.org/miccai-2024/supp/2579_supp.pdf
Link to the Code Repository
https://github.com/bitMyron/sa-swin
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Ma_Symmetry_MICCAI2024,
author = { Ma, Yang and Wang, Dongang and Liu, Peilin and Masters, Lynette and Barnett, Michael and Cai, Weidong and Wang, Chenyu},
title = { { Symmetry Awareness Encoded Deep Learning Framework for Brain Imaging Analysis } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15012},
month = {October},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes a symmetry awareness encoded deep learning framework for brain image analysis by leveraging the anatomical symmetry of the brain structure, where a symmetry-aware cross attention (SACA) module is proposed to encode symmetrical features of left and right hemispheres. In addition, the symmetry-aware pre- training proxy tasks are designed. The proposed framework is evaluated on both classification and segmentation tasks, showing superior performance compared to several baselines.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper is generally well-organized and easy to follow. The proposed framework has a broad range of potential applications.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
The research motivation presented in Figure 1 lacks persuasive impact because subfigure (a) depicting a normal brain structure doesn’t appear to be strictly symmetrical. Additionally, providing only one qualitative example may not be very convincing. It would be more beneficial to provide a quantitative statistical discussion using metrics like SSIM for various types of brain structures (normal, tumor, Alzherimer’s Disease, focal epilepsy) in the dataset.
-
The innovative aspects of the paper are not particularly prominent. It would be helpful to enumerate the differences between your approach and the cross-attention mechanism proposed by Chen et al.[z] Crossvit: Cross-attention multi-scale vision transformer for image classification[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 357-366.
-
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
-
There seems to be a mismatch between the variable X in the equation 3 and its subsequent explanation in the text.
-
In the experimental section, the downstream tasks such as classification are only compared with Swin MLP. It would be beneficial to include comparisons with other methods like CrossViT to provide a more comprehensive evaluation. Additionally, regarding the segmentation tasks, if nnUNET, TransBTS, and SwinUNETR have pre-trained weights, it would be interesting to investigate their performance with these weights.
-
Is the Swin MLP model trained on the same pretraining dataset as the method proposed in this paper? If not, the performance comparison in Tables 1 and 2 would be unfair. Furthermore, the paper lacks an explicit description of the hyperparameters used in training the Swin MLP model. Are these hyperparameters aligned with the ones employed in proposed method or the original research?
-
The experimental setup in Table 3 is not very easy to comprehend. How were the experimental results obtained by solely using the SACA module?
-
Some symbol abbreviations are used without being defined, such as MS and AUC.
-
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The experimental results and the baseline comparisons in my opinion are insufficient to draw the conclusion.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
- A Symmetry-Aware Cross-Attention (SACA) module was introduced, which leverages a cross-attention mechanism to analyze the relationship between an image and its symmetrical counterpart, facilitating a deeper understanding of brain anatomy and its inherent symmetries.
- The proposed network is pretrained on a symmetry-aware self-supervised process and can be further applied in real-world clinical datasets and analysis tasks, including classification and segmentation for multiple modalities.
- A vast dataset spanning MRI and CT modalities has been curated, and the proposed approach has undergone extensive testing and demonstrated state-of-the-art performance on diagnosis and segmentation tasks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors have extended the Swin-Unetr idea to add symmetry loss to have a symmetry aware cross attention module.
- The model has evaluated on the two large dataset with different modalities and different body parts (head and brain).
- Proper ablation studies were conducted to show that the proposed component helps in the down stream tasks.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Limited Novelty: The authors has extended the idea from Swin-Unetr (self-supervised learning) by adding a symmetry loss which aims to foster deep symmetrical feature alignment and referencing. The idea is similar to rotational augmentation , just that the rotation is done 90 degrees.
- The authors need to compare with an out-of-distribution dataset such as BTCV dataset mentioned in the Swin-UNETR paper so that we can see how effective is the SACA module.
- I am not convinced by the symmetry augmentation such as flip and rotational augmentation. The rotational augmentation should include flip as well. The authors need to explain how rotation augmentation is different from flip.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The authors need to compare with an out-of-distribution dataset such as BTCV dataset mentioned in the Swin-UNETR paper so that we can see how effective is the SACA module.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Reject — could be rejected, dependent on rebuttal (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I am not convinced by the symmetry augmentation such as flip and rotational augmentation. The rotational augmentation should include flip as well. The authors need to explain how rotation augmentation is different from flip.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
Weak Accept — could be accepted, dependent on rebuttal (4)
- [Post rebuttal] Please justify your decision
I had my concern regarding the data augmentation that was used as the important contribution and I feel the authors response convinces me. Even though the Idea is not completely novel, the contribution is still worthy for MICCAI. I upgraded my opinion to weak accept.
Review #3
- Please describe the contribution of the paper
This paper uses the transformer to encode symmetric information into the neural network. With a new added loss term, the neural network can discover symmetric properties. It can reach higher accuracy compared with its previous structure. Moreover, the transformer can reduce the labeled data during training. The results are verified for both CT and MRI.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The idea of using the symmetric structure in the brain is innovative and reasonable. Compared with previous work such as Swin MLP ([26] in the manuscript’s citation), the new structure has higher accuracy for both MRI and CT, regardless of the fine parameter tuning. This shows the advantages of encoding symmetric during training.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
This paper is interesting and well-written. I have two minor issues. (1) Practically, not all asymmetrical means disease. For example, the vitamin E capsules sometimes used in MRI scans could lead to asymmetric structure in the background. So, I wonder if a symmetric awareness network will generate overfitting or if these need to be included during training to prevent overfitting. (2) Typing issue. Missing period on page 6, line 3.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The dataset is clearly cited and hyperparameters of the neural network are listed, I believe the reproducibility is guaranteed.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
Despite the two things mentioned in the weakness section, I don’t have other comments.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Strong Accept — must be accepted due to excellence (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper is well-written, and the idea is new and reasonable. Similar papers were published in the MICCAI for breast cancer such as [DisAsymNet: Disentanglement of Asymmetrical Abnormality on Bilateral Mammograms Using Self-adversarial Learning].
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
We are grateful for the thorough reviews and appreciate the recognition of our contributions to encoding symmetric information into neural networks for brain image analysis across both CT and MRI modalities. We are committed to addressing all the reviewers’ concerns in the final version with codes published. We respond to the major points raised by the reviewers below. [To R1] Q1* Asymmetry in Brain Structures. A1* We wish to clarify that the human brain is not strictly symmetrical, and our primary motivation is not to model purely symmetry vs asymmetry but to model physical and pathological features with symmetry as a key characteristic as studied in [18]. Our motivation has been proven beneficial not just from qualitative visualization but also from quantitative studies demonstrated in [14] and [21]. Thanks for the suggestion of SSIM and similar metrics, which we will consider in the extended version. Q2* Comparison with CrossViT. A2* Our work emphasizes the development of a training framework to encode symmetry awareness rather than primarily enhancing the network structure. This framework is, therefore, adaptable to various encoders, including CrossViT. We opted for the SOTA Swin encoder to test our framework because of its proven efficacy and performance. Besides, the cross-attention mechanism is also different in our work, which is intended to model contrastive features from symmetrical counterparts by applying token-level all-to-all attention for both classification and segmentation, while CrossViT only fused the features of CLS tokens across the large-patch and small-patch branches for classification. Q3* Experimental Setups. A3* We apologize for any confusion caused by the descriptions in our paper. For classification tasks, the Swin MLP model was pretrained using the same datasets and hyperparameters as our proposed method, ensuring a fair comparison. For segmentation tasks (Table 3), we wanted to emphasize the performance of individual modules (SAH and SACA), so all methods were compared after training on the target dataset. In all experiments, we compared frameworks based on the same encoder network to maintain fairness, and we agree that it is interesting to see the performance with other network structures and pre-trained weights, which will be explored in the extended version. [To R4] Q4Asymmetry and Overfitting A4 Thank you for your insightful comment. In our study, we have implemented skull-stripping processes to mitigate the influence of extraneous factors. This ensures that our analysis is focused exclusively on brains. Besides, we agree that not all asymmetry means disease. The model is, therefore, pretrained based on distinguishing healthy and unhealthy brains with symmetry as the key feature, where physiological and pathological asymmetries can be distinguished to avoid overfitting purely symmetrical or asymmetrical features. [To R5] Q5* Is Symmetry an Augmentation? A5* We would like to clarify that our approach does not treat symmetry as an augmentation. Instead, we extract pairs of patches from anatomically symmetrically opposing locations in the original images, which are not generated via augmentation. Our framework integrates the information of pairs of patches by utilizing novel SAH and SACA. This framework fundamentally differs from the one used in SwinUNETR, where rotation is merely applied to a single input patch for contrastive learning. Q6* Out-of-Distribution (OOD) Testing. A6* Our testing comprehensively includes OOD datasets to evaluate the robustness and effectiveness of our framework. Table S1 details the sources of all datasets used in our experiments, which are collected from multiple clinical centers with varied settings, ensuring that the test conditions are rigorously OOD. Besides, the suggested BTCV data is not directly relevant to our work, as it only contains abdomen CTs, which diverges significantly from our research aimed at disease diagnosis through brain structure analysis.
Meta-Review
Meta-review #1
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
N/A
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
N/A
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’
The proposed approach encodes brain asymmetry to enhance feature learning for brain image analysis. Overall, it is an interesting approach that is relevant to the neuroimaging community, which has been investigating brain asymmetry as a potential biomarker in various contexts. After rebuttal, R5 has increased the rating.
- What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).
The proposed approach encodes brain asymmetry to enhance feature learning for brain image analysis. Overall, it is an interesting approach that is relevant to the neuroimaging community, which has been investigating brain asymmetry as a potential biomarker in various contexts. After rebuttal, R5 has increased the rating.