Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Semi-supervised cross-domain segmentation, also referred to as Semi-supervised domain adaptation (SSDA), aims to bridge the domain gap and enhance model performance on the target domain with the limited availability of labeled target samples, lots of unlabeled target samples, and a substantial amount of labeled source samples. However, current SSDA approaches still face challenges in attaining consistent alignment across domains and adequately addressing the segmentation performance for the tail class. In this work, we develop class-aware mutual mixup with triple alignments (CMMTA) for semi-supervised cross-domain segmentation. Specifically, we first propose a class-aware mutual mixup strategy to obtain the maximal diversification of data distribution and enable the model to focus on the tail class. Then, we incorporate our class-aware mutual mixup across three distinct pathways to establish a triple consistent alignment. We further introduce cross knowledge distillation (CKD) with two parallel mean-teacher models for intra-domain and inter-domain alignment, respectively. Experimental results on two public cardiac datasets MM-WHS and MS-CMRSeg demonstrate the superiority of our proposed approach against other state-of-the-art methods under two SSDA settings.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0494_paper.pdf

SharedIt Link: https://rdcu.be/dZxda

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72111-3_7

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0494_supp.pdf

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Cai_Classaware_MICCAI2024,
        author = { Cai, Zhuotong and Xin, Jingmin and Zeng, Tianyi and Dong, Siyuan and Zheng, Nanning and Duncan, James S.},
        title = { { Class-aware Mutual Mixup with Triple Alignments for Semi-Supervised Cross-domain Segmentation } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15008},
        month = {October},
        page = {68 -- 79}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The paper proposes a class-aware mutual mixup strategy with triple alignment for semi-supervised cross-domain segmentation. The mutual mixup obtains the maximal data distribution diversification and enables the model to focus on the tail class during cross-domain alignment. A a triple consistent alignment is established, and a cross-knowledge distillation is further introduced for intra-domain and inter-domain alignment.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well-organized and easy to follow. The research problem of class imbalance in semi-supervised cross-domain segmentation is well-motivated. The class-aware mutual mixup strategy that extends one-directional consistency to a mutual mixing approach is somewhat innovative.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    (1) The impact of using the “class-aware mixup strategy that prioritized the tail categories” on the segmentation ability of head categories has not been considered, and it is questionable whether the segmentation ability of head categories has been weakened. (2) At the end of Sec.2.1, the sampling method of the class-aware mixup strategy is introduced, but the combination of this sampling method and the class-aware mixup method is not clearly explained. The “class-aware mutual mixup” process in Fig.3 does not reflect this sampling process. (3) The two datasets used in the experiments are paired and unpaired, but the paper does not elaborate on the impact of data pairing on the proposed method. It is unknown whether the unpaired data would weaken the ability of the method.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. Doubts have been raised about whether the proposed method weakens the segmentation ability of head categories. Please provide further analysis.
    2. At the end of Sec.2.1, the combination of sampling methods and the class-aware mixup method should be explained. The characterization of the sampling strategy should be included in Fig.3.
    3. The effect of data pairing on the method should be accounted for in the experimental analysis.
    4. The layout of the “class-aware mutual mixup” in Fig.3 should be modified. To accommodate reading comfort, it is recommended to change the direction of the output from top to bottom. The current layout makes it hard to find the inputs and outputs.
    5. There are typos in Figure 3, where “X_m (S, U)” is written as “X_m (S, L)”.
    6. The font in all the images in the text is too small to read.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper proposes a novel class-aware mutual mixup strategy to obtain the maximal diversification of data distribution and enable the model to focus on the tail class. Overall, the topic is interesting, and the work is generally acceptable. However, several points can still be drawn to improve this paper further, and suggestions for improvement have been proposed in point 10.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    The authors tackle the domain shift problem in segmentation models by introducing Class-Aware Mutual Mixup with Triple Alignments (CMMTA), a new semi-supervised domain adaptation technique. CMMTA incorporates class-aware mutual mixup across three pathways: between the source domain and labeled target domain, between the source domain and unlabeled target domain, and within the target domain. They also propose a class-aware mixup strategy focusing on low-frequency labels, emphasizing classes at the tail of the distribution. Evaluation on cardiac segmentation shows that CMMTA outperforms previous approaches in two semi-supervised domain adaptation scenarios.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The problem addressed is not novel but remains crucial for the successful application of automatic segmentation in medical practice. Accurate models must account for large data variations encountered at test time, beyond the domain data they are trained on.
    • The proposed formulations are straightforward, intuitive, and clearly presented, making them easy to follow.
    • The evaluation compares the proposed model against multiple existing techniques in two settings, employing a comprehensive set of metrics. Ablation studies are also conducted to validate the improvements made by the proposed modifications.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The components of the proposed model are primarily incremental modifications of existing approaches, many of which are known to achieve better results in other contexts of image segmentation. Therefore, the paper has limited technical novelty. However, demonstrating that these modifications together result in a better model, as shown in their experiments and ablation studies, is valuable to the community.
    • All experiments were conducted solely on a single segmentation backbone, DeepLab-V2, which is considered somewhat outdated. It remains unclear whether the improvements demonstrated in the paper would also be achieved when applied to other backbones, such as modern transformers.
  • Please rate the clarity and organization of this paper

    Very Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    I believe these domain adaptation techniques need to prove successful across various segmentation models to be truly valuable. They should be model-agnostic. However, while the proposed techniques do not make any hard assumptions about their backbone, no results are shown using anything other than the outdated DeepLabV2. The paper could significantly benefit from experiments showcasing the effectiveness of these techniques across different segmentation models.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    This paper addresses a crucial problem in medical applications by proposing a model for semi-supervised domain adaptation. The proposed model effectively combines existing techniques, as demonstrated in thorough experiments and ablation studies. However, its dependence on the outdated DeepLabV2 backbone restricts the generalizability of the findings. In my view, considering the significance of the problem and the effectiveness of the proposed techniques, the paper could be accepted with some reservations.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #3

  • Please describe the contribution of the paper

    The paper presents a method for semi-supervised cross-domain segmentation. It combines a class-aware mutual mixup method that combines mixup of intra-domain semi-supervised data, with mixup of inter-domain data. The method is demonstrated on two public cardiac datasets: MM-WHS (MRI/CT) and MS-CMRSeg (different MR sequences). The experiments include several settings, a comparison with alternative unsupervised and semi-supervised domain adaptation methods, and an ablation study. The proposed model outperforms the others.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    • The experiments are fairly elaborate and well-designed: different proportions of labeled/unlabeled data, a comparison with alternative methods, and an ablation study.

    • The idea to combine inter-domain and inter-domain alignment is interesting and seems to be effective.

    • The method is quite clearly described.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • There are experiments on two datasets, but they are both on cardiac segmentation tasks. It would have been interesting to have a bit more variety.

    • The model is a combination of several strategies: mutual mixup, the class-aware strategy, the CKD enhancement. This makes it slightly difficult to compare with other methods. The ablation study shows that all components contribute to the performance, but individually, the performance is worse than that of competing methods.

    • It would be nice to share the code for the experiments.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The mixup method combines data from CT and MRI. Is that a good idea? Can we expect that a single model can process both types of images at the same time?

    Why do the two datasets require different optimization strategies (SGD/Adam etc.)?

    Section 2: The past tense is confusing. Usually, the problem and method are described in the present tense. The past tense is reserved for the experiments and possibly the related work. Reading a Methodology section in the past tense is confusing: are the authors describing the method they’re proposing in this paper, or some older method they tried before?

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Slightly convoluted but interesting method. Good experiments and comparison with other methods. Results suggest that the method works fairly well.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We thank all reviewers for your suggestions and positive recognition of our work!

To Reviewer #1: Thanks! Head category: Thank you for raising this important question! Although we manually prioritize sampling from the tailed class, the input images are unpaired, leading to a high likelihood that tailed classes might still be pasted onto cropped images containing head categories. In Table 3, experiment #4, the addition of class-aware sampling significantly enhances performance both in the tail category (AA) and head categories (LAC and MYO).

Pair Data: We consistently use an unpaired data setting throughout our experiments.

Combination Explanation: We appreciate your attention to detail. We’ll add more descriptions at the end of Section 2.1 and improve Figure 3 for better clarity on the sampling strategy.

Figure Layout: Thank you for your feedback! We will revise the layout of Figure 3 to enhance readability and ensure it is easy to understand.

Typos & font: Thank you for pointing this out. We will thoroughly check for typos and revise any unreasonable fonts in the later version.

To Reviewer #3: Thanks! Datasets: Thank you for your understanding. Due to the page limitations of MICCAI, we could only conduct experiments on two datasets. However, we plan to verify our method on additional SSDA tasks in a future extended version.

Code: Thank you! We are going to release the code later.

Combining modalities: Thank you for your feedback! Our paper approaches the alignment of domains from the perspective of data augmentation. CT and MRI mixup is a direct method to generate inputs that encompass both domain distributions, effectively bridging the domain gap. This technique has been widely validated in general computer vision tasks [5,11], with real and game images mixup. Our ultimate goal is to develop a unified model capable of processing different modality images with high performance simultaneously.

Optimizers: Thank you for raising this question! We referred to previous methods [17, #1] on these two datasets to select the appropriate optimizer. Through our experiments, we found that SGD achieved better results on the MM-WHS dataset, while Adam yielded better results on the MS-CMRSeg dataset. Ref: #1. Fuping, Wu and Xiahai, Zhuang. “Unsupervised domain adaptation with variational approximation for cardiac segmentation.” TMI (2021).

Writing: We appreciate your advice. We will carefully review and correct the time tense throughout the entire paper, with particular attention to the method and experiment sections.

To Reviewer #4: Thanks! Novelty: Thank you for your feedback. In our approach to SSDA, we address the limitation of one-way mixup among the source domain, labeled target domain, and unlabeled target domain by employing a two-way mixup with triple alignment. This maximally diversifies the data distribution. Additionally, we introduce a simple yet effective class-aware sampling strategy to improve performance on the tail class. Unlike the shared weight model and co-teaching model, we propose cross-knowledge distillation to further enhance model generalization.

Backbone: Thank you for raising this question! We acknowledge the importance of proving the model-agnostic nature of our method. In the future, we plan to verify our approach on an improved transformer architecture specifically tailored for the SSDA task. Our prior investigation indicates that U-Net and DeepLab still achieve state-of-the-art performance for segmentation during UDA task. However, the transformer backbones referenced in [#1] yielded less competitive results than the base backbone on the MM-WHS dataset. Therefore, for this study, we focused solely on our method based on the CNN backbone. Moving forward, we aim to design a competitive transformer architecture for transfer learning tasks. Ref: #1. Ji, Wen, and Albert CS Chung. “Unsupervised Domain Adaptation for Medical Image Segmentation Using Transformer With Meta Attention.” TMI (2023).



Meta-Review

Meta-review not available, early accepted paper.



back to top