Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Implicit functions have significantly advanced shape modeling in diverse fields. Yet, their application within medical imaging often overlooks the intricate interrelations among various anatomical structures, a consideration crucial for accurately modeling complex multi-part structures like the heart. This study presents ImHeart, a latent variable model specifically designed to model complex heart structures. Leveraging the power of learnable templates, ImHeart adeptly captures the nuanced relationships between multiple heart components using a unified deformation field and introduces an implicit registration technique to manage the pose variability in medical data. Built on WHS3D dataset of 140 refined whole-heart structures, ImHeart delivers superior reconstruction accuracy and anatomical fidelity. Moreover, we demonstrate the ImHeart can significantly improve heart segmentation from multi-center MRI scans through a retraining pipeline, adeptly navigating the domain gaps inherent to such data.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0411_paper.pdf

SharedIt Link: pending

SpringerLink (DOI): pending

Supplementary Material: N/A

Link to the Code Repository

N/A

Link to the Dataset(s)

https://cvlab.epfl.ch/data

BibTex

@InProceedings{Yan_Generating_MICCAI2024,
        author = { Yang, Jiancheng and Sedykh, Ekaterina and Adhinarta, Jason Ken and Le, Hieu and Fua, Pascal},
        title = { { Generating Anatomically Accurate Heart Structures via Neural Implicit Fields } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15001},
        month = {October},
        page = {pending}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    The authors propose ImHeart, a model for the 3D representation of heart structures from MRI scans. The method models 8 components of the human heart from MRI scans by employing a latent variable model. This model relies on multi-class implicit functions and considers the complex interrelationship between different heart parts. The authors compiled their own dataset of whole-heart structures (WHS3D) and demonstrated the application of their model for enhancing the segmentation of real MRI scans with lower resolution and blurring.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The main strength of the paper lies in the complexity of the task the authors address: model a structure of heart, considering the interrelationship between its parts. The authors propose a framework consisting of three auto-decoders (ConvONet) for unified deformation (representing relationships between parts), registration (handling pose variability), and residual (fine-tuning). The proposed method extends ImplicitAtlas by modeling multi-class shapes.

    The author demonstrates the application of the generated models for enhancing the MRI images to be used for re-training a heart segmentation model.

    The authors explain how their latent variable model, based on learnable templates, differs from the state-of-the-art by employing multi-class implicit functions instead of single-class ones and considering multi-versus-pairwise connectivity. Despite being theoretically less precise, their model maintains anatomical precision.

    To train their model, the authors compiled their own dataset of 140 whole-heart structures. Moreover, they demonstrated the application of their model for enhancing the segmentation of real MRI scans.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The results of modeling and segmentation are presented, but details such as whether the hearts were healthy, the diversity of the patients, etc., are not specified, which is crucial for readers to evaluate the results effectively.

    The structure of the paper appears somewhat chaotic and difficult to follow. The introduction contains a detailed description of methods and results, while the methods section discusses related works and experiments cover the evaluation metrics. Some statements, such as “can be realized in various manners” or “can be learned through various methods,” lack support from references.

    The methods section transitions abruptly from detailed formulas to superficial explanations of implementation details. Additionally, the authors use implicit registration techniques involving affine transformations with rotation and shift but do not explain why scaling was excluded.

    No data on training performance or statistical analysis is provided.

    The figure descriptions are unclear without reading the corresponding chapter of the paper.

  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    The authors do not provide the source code of their implementation, which could hinder reproducibility. However, they state that the dataset of heart shapes they compiled will be made public upon acceptance.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    I believe the authors have done an excellent job addressing a very complex task and achieving good results, both in modeling and in the application on segmentation.

    However, I think there is room for improvement in the structure of the paper to enhance its comprehension. Shortening the introduction and preliminaries could help alleviate the limitations imposed by page constraints and allow for a more detailed discussion of used datasets, future work and the limitations of the current work.

    Providing more details on the diversity of the heart dataset is crucial for readers to evaluate the results effectively, therefore should be added.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    While the work appears promising, I find it challenging to evaluate the results without a comprehensive understanding of the dataset’s diversity used for demonstration. The description of the methods is difficult to follow, indicating the need for a significant revision. The authors may need to decide whether to prioritize a thorough description of the methods, or to focus more on the results and application.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The authors demonstrated an understanding of the main issues and clarified the questionable points, promising to address them where possible. I recommend acceptance after rebuttal.



Review #2

  • Please describe the contribution of the paper

    The authors introduced in this paper ImHeart: a method based on neural implicit fields to model 8 different structures of the heart. ImHeart extends template-based implicit approaches by adapting them to several classes and adds an implicit registration technique to handle the variability of poses in medical data. They demonstrate superior performance in modelling the heart shape (using the Volumetric Dice Score and a proposed metric: Number of connected components) and show potential benefits of the method as a retraining strategy able to overcome domain gaps.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper is clear and well-written
    • Meaningful evaluation of the method’s performance (comparison with methods with or w/o template and ablation study conducted of the importance of each part of the model (i.e Sep/ Uni deformation fields and with or without implicit registration))
    • Introduction of a relevant metric to evaluate the performance of their method.
    • Shows the benefit of the approach for a segmentation on two external datasets, including one with a large domain gap
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • Report of the standard deviation: in the quantitative evaluation, you should report the standard deviation when applicable

    • Lack of experimental setup for reproducibility

    Minor :

    • Refine annotation quality in Section 3.1: o The authors mentioned “several empirical rules”. For the sake of reproducibility, can you provide more information about these rules? Maybe in Supplementary details if the space is lacking. o Did you evaluate the two steps independently, i.e. the Neural Annotation Refinement alone or the empirical rules alone? An illustration in Fig. 2 or generated sample in supplementary material will be relevant with only one of the techniques will be relevant.

    • The author claimed to introduce an implicit registration technique. [1] describes a similar concept. “Include an implicit registration” seems a more appropriate phrasing.

    [1] Park, Keunhong, et al. “Nerfies: Deformable neural radiance fields.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    • In the abstract, CAD is not defined and no acronym should be used
    • Misspelling : o Section 4.1, 3rd paragraph: multple classes (an « i » is missing) o Title Table 1: registration
    • Definition of the metric: how did you define a connected component in the metric?
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The authors proposed a well-designed extension of Neural implicit template approach and demonstrated via a proper evaluation its benefit. In particular, the method shows good capability as a retraining pipeline to improve segmentation performance on datasets with domain gaps.

  • Reviewer confidence

    Not confident (1)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Accept — should be accepted, independent of rebuttal (5)

  • [Post rebuttal] Please justify your decision

    The authors addressed all of my main comments. Therefore, I have raised my grade from “Weak Accept” to “Accept”



Review #3

  • Please describe the contribution of the paper

    This study introduces ImHeart, a latent variable model that integrates learnable templates with a unified deformation field to model complex heart structures. The model is validated on the WHS3D dataset consisting of 140 whole-heart structures and is shown to improve heart segmentation accuracy in multi-centre MRI scans through a retraining pipeline. ImHeart addresses the challenge of capturing the interrelations among various heart components and manages pose variability in medical imaging data.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper introduces a novel approach by integrating learnable templates within a unified deformation field to capture interrelations among various heart structures.
    • The study employs an innovative retraining pipeline that utilises initial segmentations from multi-centre MRI scans as input for the ImHeart model, which refines these segmentations.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • The incremental nature of the improvements suggests that the proposed approach may offer only marginal benefits over current methodologies, necessitating a clearer demonstration of substantial advancements.
    • The results presented in the paper lack rigorous statistical analysis, such as paired t-tests or confidence intervals, which are necessary to substantiate claims of significant improvement.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    1. It is suggested to include the mean Contrast-to-Noise Ratio (CNR) for both MMWHS-MRI and In-House datasets to substantiate the statement regarding the lower resolution of In-House datasets.

    2. The inclusion of the Average Symmetric Surface Distance (ASSD) metric, as evaluated in [6], would provide a more quantitative complement to the topological assessments currently discussed.

    3. Improvements to Table 1 could be made by clarifying that the analysis pertains to the test set, including the number of samples in this set. Adding standard deviations to each mean measure could enhance interpretability. Furthermore, the last three rows could benefit from a title such as “Proposed” to distinguish these results. It may be helpful to replicate the DSC value of 0.939 for single-class analysis for both separate (Sep) and unified (Uni) configurations for clarity, assuming the values are identical.

    4. Performance comparisons should be substantiated with statistical tests, such as paired t-tests, to rigorously evaluate the improvements and substantiate claims of significant enhancement.

    5. In the downstream application section, consider training the segmentation model on both datasets to establish a comprehensive baseline performance. Given the inherent distribution drift, if the proposed model can achieve this baseline performance without requiring manual annotations from out-of-distribution samples—merely by correcting predicted ones—the utility of the model would be more convincingly demonstrated.

    6. The potential of ImHeart to map the time-dependence of cardiac functions should be discussed, considering its relevance to dynamic cardiac studies.

    7. For ease of verification, it is recommended to order the references as they appear in the text, rather than alphabetically.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The recommendation of “weak accept” is primarily due to the novel integration of learnable templates with a unified deformation field in the ImHeart model, which is a promising approach to addressing complex interrelations among heart structures in medical imaging. Additionally, the innovative potential use of a retraining pipeline to reduce domain gaps in multi-centre MRI scans demonstrates potential clinical applicability. However, the recommendation is tempered by significant weaknesses: the paper does not provide rigorous statistical validation of its results, the reported improvements over existing methods are not substantiated with sufficient quantitative analysis, and there is a notable lack of discussion regarding the limitations and potential biases of the methodology.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The reviewers’ comments have been addressed satisfactorily.



Review #4

  • Please describe the contribution of the paper
    • The study introduces “ImHeart,” a latent variable model designed to model complex heart structures with high fidelity. This model leverages learnable templates to capture the nuanced relationships among various heart components using a unified deformation field.
    • An implicit registration technique to address the variability in poses frequently observed in the collection of medical data.
    • ImHeart achieves superior reconstruction accuracy and anatomical precision compared to previous methods.
    • ImHeart improves heart segmentation from multi-center MRI scans. By retraining a segmentation network with outputs refined by ImHeart, the model adeptly navigates the domain gaps often present in data from various centers, leading to improved segmentation accuracy.
  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Implicit Registration
    • Superior performance for Heart shape reconstruction compared to other baselines.
    • Strong performance improvement using ImHeart as assessed by multiple metrics, on two datasets with significant domain gap - test (MMWHS-MRI) and external validation (In-House) datasets for segmentation.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • Not all empirical rules mentioned in the paper. Could have used the supplementary materials to inform about the rules, for further validity.
    • small size of test data for Heart shape modelling, as well as lack of external dataset.
  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Not reproducible, given the empirical rules not specified here - “…we applied several empirical rules aimed at ensuring the topological accuracy of the labels…”

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Good overall paper. Strong results for segmentation and shape reconstruction tasks

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
    • Novel method for heart structure generation.
    • Strong results for 2 tasks.
    • Significantly better results than previous methods for Heart shape modeling task.
    • Good validation for segmentation task on an small sized external data which is of different domain from training and test dataset.
  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We appreciate the high-quality reviews from the reviewers (R1, R3, R4, R5) and their acknowledgment of our strengths in multi-center evaluation (R1, R3, R5), good results (R3, R4), method novelty (R3, R5), writing (R1) and task challenges (R4). We address their key concerns as follows.

  1. R1, R4, R5: Statistical Analysis. Due to the new rule in this year’s Rebuttal Guide, we can only provide further details on the results already in the paper. All results in the paper were averaged over 3 experiments. Regarding standard deviation (R1, R5), the std for DSC in Tab 1 and 2 ranges from 0.003 to 0.011, and for CC, the std ranges from 0 to 0.14. For the t-test (R5), we calculated that the unified deformation is significantly better than the separate one (p=0.0127), and implicit registration is also significantly effective (p=0.0162).

  2. R1, R3: Empirical Rules for Repairing Labels / Reproducibility. We will add details of these empirical rules in the final version, while we emphasize that they are used only to generate the WHS3D shape ground truth to ensure topological correctness. These rules are not used in other parts of the method. Given that WHS3D data will be open-sourced, we believe this will not affect the reproducibility.
    The rules include several morphological operations to ensure that the 8 classes each have only 1 connected component (CC). If not, we retain the largest CC after a closing operation. Additionally, we ensure connectivity between classes: ventricle vs atrium, aorta vs left ventricle, and right ventricle vs pulmonary artery. Since these labels come from real data, disconnected cases are rare (usually due to annotation noise), and we slightly dilate to ensure proper connection.

  3. R3: Small Data Size.
    Our goal is to establish a latent variable model with rich cardiac structures, and there is very little fully annotated data publicly available. This is a limitation. To overcome it, we are extending our approach by using partial-label learning on more heterogeneous datasets.

  4. R4: Patient Details.
    The source data for WHS3D comes from previous studies [12, 18, 21], primarily past MICCAI challenges. Unfortunately, these papers also do not disclose details about the patient cohorts, and we only know that there are no large anatomical variations among them. We will mention this as a limitation.

  5. R4: Writing. We will “shorten the introduction and preliminaries” to allow more space for discussing “datasets, future work, and limitations”. Additionally, we will add references for “can be realized in various manners” [19, 25, 28] and “can be learned through various methods” [15, 16].

  6. R4: Why Scaling Was Excluded in Implicit Registration.
    The matrix R in A=(R,b) is a freely optimized 3x3 matrix, and not a rotation matrix as erroneously stated. Our apologies for the typo; it will be fixed. This means our method actually includes scaling and shearing.

  7. R5: More Metrics. Due to the Rebuttal Guide, we cannot provide these metrics in the rebuttal. However, we will include CNR in Fig 3 to highlight the dataset differences, and also ASSD from [6] in Tab 1 and 2 to better differentiate topological assessments.

  8. R5: Manual Annotations on OOD Samples.
    We think there is a misunderstanding. In the OOD in-house dataset, we do not use manual annotations for training; instead, it operates solely “by correcting the predicted ones”.

  9. R5: Cardiac Dynamics. We completely agree that the deformed template is well-suited for dynamic cardiac studies. We are considering exploring this direction.

Other Issues:
A. R1: Nerfies. We will discuss the relationship between Nerfies and Implicit Registration. B. R4: Training Performance. Considering space limitations and the limited significance of training results, we cannot guarantee reporting training performance in the final version. C. R5: References. We will order the references as they appear. D. R1, R4, R5: Typos / Wording. We will fix these issues.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



back to top