List of Papers Browse by Subject Areas Author List
Abstract
Our method solves unified multi-modal learning in an diverse and imbalanced setting, which are the key features of medical modalities compared with the extensively-studied ones. Different from existing works that assumed fixed or maximum number of modalities for multi-modal learning, our model not only manages any missing scenarios but is also capable of handling new modalities and unseen combinations. We argue that, the key towards this any combination model is the proper design of alignment, which should guarantee both modality invariance across diverse inputs and effective modeling of complementarities within the unified metric space. Instead of exact cross-modal alignment, we propose to decouple these two functions into representation-level and task-level alignment, which we empirically show is both dispensable in this task. Moreover, we introduce tunable modality-agnostic Transformer to unify the representation learning process, which significantly reduces modality-specific parameters and enhances the scalability of our model.
The experiments have shown that the proposed method enables one single model handling all possible combinations of the six seen modalities and two new modalities in Alzheimer’s Disease diagnosis, with superior performance on longer combinations.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1760_paper.pdf
SharedIt Link: https://rdcu.be/dV1WI
SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72384-1_46
Supplementary Material: N/A
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{Fen_Unified_MICCAI2024,
author = { Feng, Yidan and Gao, Bingchen and Deng, Sen and Qiu, Anqi and Qin, Jing},
title = { { Unified Multi-Modal Learning for Any Modality Combinations in Alzheimer’s Disease Diagnosis } },
booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
year = {2024},
publisher = {Springer Nature Switzerland},
volume = {LNCS 15003},
month = {October},
page = {487 -- 497}
}
Reviews
Review #1
- Please describe the contribution of the paper
The present work proposes a novel method for the Alzheimer’s disease diagnosis based on a unified multi-modal learning, which aims at managing missing scenarios and at handling new modalities and unseen combinations. In details, the authors employed a Transformer approach to easily adapt the architecture to new modalities.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength is related to the capability of the proposed architecture to deal with missing data and multi-modality data. Furthermore, the tunability of model-agnostic Transformer reduce the modality-specific parameters assuring training stability and minimizing overfitting.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weakness I found is related to the poor interpretability of the architecture and the findings, which prevent the translation into a clinical setting.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
The experimental part should be improved to ensure the reproducibility. Sample size per diagnosis is missing, as well as descriptive statistics of demographics and clinical data.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
The manuscript is well written, and the description of the architecture is highly detailed. I have only two minor suggestions. The first is to improve the abstract, because it is not clearly stated the main aims of the work. The second is to add a section with Discussion that also reports the main limitation of the study.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although I appreciated the authors’ effort to propose a robust and stable architecture, I recommended weak acceptation for the poor clinical feasibility of the study.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #2
- Please describe the contribution of the paper
The paper introduces a tunable modality agnostic Transformer to reduce parameters. At the same time it allows to combine various and mixing of modalities in the context of Alzheimer’s Disease diagnosis. It further allows the integration of new unseen modalities during the inference phase.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Introduction of the modality agnostic transformer and a new alignment method between the different modalities. In contrast to previous work their approach allows to keep differences between modalities instead of only maximizing the similarities.
- They perform extensive experiments on a Alzheimer’s Disease database with eight different modalities.
- The ablations show that their approach performs increasingly better with an increasing number of added modalities.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- In depth explanation of how to update the model for new modalities is missing. It would be interesting to see here how many data samples are necessary or if this process is performed unsupervised.
- It would be interesting to see the influence of the different modalities to the final prediction. Minor:
- The short title of the paper is missing.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
N/A
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
It would be important to see how the training dataset is defined exacatly. How much overlap is there between the individual modalities etc. On page 4 the authors mention ‘AnyMod’ which I assume is supposed to be the name of their architecture. However, it is not introduced or mentioned elsewhere in the text.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Weak Accept — could be accepted, dependent on rebuttal (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Overall the paper is concise and shows a very interesting approach for multi-modal classification or diagnosis in the context of Alzheimer’s Disease. However, in depth explanation about the dataset and updating for unseen modalities is missing.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Review #3
- Please describe the contribution of the paper
This paper describes an architecture for multimodal learning which is parameter efficient and robust to missing or even new modalities. This is achieved by using a common component for projection of the different modalities to the task space, and to the introduction of multiple anchors in the task space.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Relevant topic, it often occurs that not all modalities are present for all cases. Ablation study seems thorough, even if a bit hard to understand in some parts.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The results aren’t as compelling as they could be, as the MSP is a clear winner in Table 1. This is justified because that approach has 7 times as many parameters, but it might have been stronger to just increase the number of parameters in the authors approach to match.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Do you have any additional comments regarding the paper’s reproducibility?
Public dataset is used, but source code is not mentioned.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
P 7 Our experiments has – Our experiments have Sec 3.4 paramters -parameters Table 2 - it is hard to understand what the Mean columns are - they are not the mean of the other table entries or else there is a typo Fig 3 might be easier to understand and write about if each sub figure was individually identified like Fig 3 a i, Fig 3 c ii The layout of this figure is quite crammed.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making
Accept — should be accepted, independent of rebuttal (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Topic is relevant and generalizes beyond the specific example presented. Paper is well focussed and clear.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed
N/A
- [Post rebuttal] Please justify your decision
N/A
Author Feedback
N/A
Meta-Review
Meta-review not available, early accepted paper.