Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Accurate segmentation of the aortic valve (AV) in computed tomography (CT) scans is crucial for assessing AV disease severity and identifying patients who may benefit from interventional treatments, such as surgical and percutaneous procedures. Evaluation of AV calcium score on non-contrast CT scans emphasizes the importance of identifying AV from these scans. However, it is not a trivial task due to the extremely low visibility of AV in this type of medical images. In this paper, we propose a method for semi-automatic generation of Ground Truth (GT) data for this problem based on image registration. In a weakly-supervised learning process, we train neural network models capable of accurate segmentation of AV based exclusively on non-contrast CT scans. We also present a novel approach for the evaluation of segmentation accuracy, based on per-patient, rigid registration of masks segmented in contrast and non-contrast images. Evaluation on an open-source dataset demonstrates that our model can identify AV with a mean error of less than 1 mm, suggesting significant potential for clinical application. In particular, the model can be used to enhance end-to-end deep learning approaches for AV calcium scoring by offering substantial accuracy improvements and increasing the explainability. Furthermore, it contributes to lowering the rate of false positives in coronary artery calcium scoring through the meticulous exclusion of aortic root calcifications.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/3643_paper.pdf

SharedIt Link: https://rdcu.be/dV51M

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72114-4_55

Supplementary Material: N/A

Link to the Code Repository

N/A

Link to the Dataset(s)

https://doi.org/10.5281/zenodo.12672626

BibTex

@InProceedings{Buj_Seeing_MICCAI2024,
        author = { Bujny, Mariusz and Jesionek, Katarzyna and Nalepa, Jakub and Bartczak, Tomasz and Miszalski-Jamka, Karol and Kostur, Marcin},
        title = { { Seeing the Invisible: On Aortic Valve Reconstruction in Non-Contrast CT } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15009},
        month = {October},
        page = {572 -- 581}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    Precisely segmenting the aortic valve is important for computer-aided diagnosis and treatment. Nevertheless, its limited visibility presents a formidable challenge in non-contrast CT. In this study, the authors introduces a method for semi-automatically generating ground truth and a weakly-supervised model for aortic valve extraction.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Aortic valve segmentation in non-contrast CT images is challenging and has significant clinical value.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    1.The novelty of the proposed method is limited. The main componants in the proposed strategy including registration and segmentation used the mature algorithms such as image registation method in [26,11] and nn-UNet[8] etc, without any major innovative designs. 

    1. The author claimed that the main motivation to developing the method for aortic valve segmentation is to evaluate AV calcification. However, the dataset for algorithm evaluation is orCaScore challenge dataset which is used for coronary calcium scoring not for AV calcification detection. 
    2. As displayed in Fig.1, the boundary of the aortic valve is unclear in the non-contrast CT images. Thus, the accuracy of the proposed mehtod could be difficult to be evaluated comprehensively. 
    3. The comparison with other SOTA methods is missing in this paper.
  • Please rate the clarity and organization of this paper

    Poor

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The writing of this paper nees to be improved. The description of the proposed methods is not clear. Some errors can be found in the paper, for example, the contratst and non-contrast captions need to change their position in the box of ‘ROI extraction’ in Fig.3.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Reject — should be rejected, independent of rebuttal (2)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Please refer the main weaknesses of the paper.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper presents the training of an nnU-net to segment the aortic valve (AV) from non-contrast enhanced CT, which is significant given the lack of visibility of the boundaries of the AV in many places without contrast. The results appear to be good, with the average mean absolute error under 1mm. The primary novelties of the paper are in the data preparation and evaluation methods, which rely on pairs of contrast and non-contrast CT scans to provide ‘ground truth’ segmentations for training and evaluation.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The use of the contrast enhanced scans to derive ground truth for training and evaluation is very clever, and could be a useful tool in other applications as well.

    The use of simple and well-established algorithms such as nnU-net for segmentation makes the approach very accessible, and also suggests interesting implications about these well-known models. For example, it is interesting that the model is able to successfully extrapolate the shape of the AV, even where the boundary is invisible, suggesting that the model has learned a shape prior in addition to boundary detection.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The human evaluation reduced the internal data set for training the main segmentation model from 268 to 143, meaning 125 (47%) of the cases were rejected by the human evaluator. These segmentations were generated by registering the contrast enhanced image to the non-contrast image, and then running a segmentation algorithm on the registered contrast image. This large number of rejections is concerning, both as a potential source of bias, and as a limitation of the scalability/efficiency of the training process.

    The high-level approach to evaluation is very nice, however, the reported metrics (only mean absolute distance) are limited. For example, the distribution of the Hausdorff distance could give a sense of the performance in the worst areas, and of course dice scores are now conventional for segmentation.

    While I think the solution is novel, the method is essentially a clever combination of well-known algorithms, somewhat limiting the overall novelty of the contribution. On the whole, I don’t think this is a bad thing, but still noteworthy.

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Do you have any additional comments regarding the paper’s reproducibility?

    A link to the source code is mentioned in the paper, but the link is not working, so I can’t verify it. Most of the system is based on open tools/algorithms and would be reproducible without code. The customized ICP is the exception to this, and the authors should ensure to release the code, because the details as written are likely not sufficient to reproduce the algorithm.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    As mentioned above, the metrics in the evaluation are quite limited. I think it would be useful, even as supplementary material, to include distributions of other metrics such as Hausdorff distance or Dice score. I don’t want the authors to have to rerun any experiments, so I wouldn’t consider this a requirement for acceptance, but if the registered segmentation meshes are still easily available, it might be valuable to include.

    Regarding the large number of rejected cases during the training phase, it would be useful if more information could be given about what went wrong. These segmentations were generated by registering the contrast and non-contrast images, then running the segmentation algorithm on the contrast-enhanced image. Are the rejections mostly due to failure of segmentation, or registration?

    Finally, the customized ICP is described in very little detail, and I don’t completely understand the footnote on 7. Is there an intuitive reason that dropping points would improve accuracy in this context, or that the size would vary between the two scans?

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I think this is a strong submission overall. The results are impressive, the method is well-explained, the data generation for training and evaluation are clever, and the paper is clearly written.

    However, I have some minor reservations about the number of rejections at the first stage, and also feel that the novelty of the method is somewhat limited.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Accept — should be accepted, independent of rebuttal (5)

  • [Post rebuttal] Please justify your decision

    My evaluation is unchanged by the rebuttal. As before, I think the paper should be accepted. Thanks to the authors for the additional clarifications they provided to my points.



Review #3

  • Please describe the contribution of the paper

    This paper presents a method for 3D reconstruction of the aortic valve from non-contrast CT images. Due to the difficulty of obtaining ground truth masks (because of the lack of contrast between the aortic valve and the rest of the heart) the images with contrast of the patient that exist in the cases, are used. From these masks and using a registration process, the masks of the aortic valve corresponding to the images without contrast are obtained. These masks (after manual evaluation) will be used for the training of a nnU-Net network, which is the one that finally allows obtaining a model for the segmentation of the aortic valve from images without contrast. The evaluation of the method is performed by comparing the surface mesh of the aortic valves obtained with and without contrast. The result is an average distance of less than 1 mm between the two.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The results obtained are undoubtedly the main strength. The median (0.763) is even better than the mean due to the presence of an outlier (1.599). Otherwise, the proposed method uses conventional techniques. Basically, a nnU-Net network. The most remarkable is the process for obtaining the ground thrust masks from the contrast images of the same case. In addition, the data comes from 2 different centers including TC from 3 different companies.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The weaknesses of the paper are those that the authors themselves include in the “Limitations and Challenges” section, namely that the proposed method has not been tested in cases with anomalies such as bicuspid prosthetic aortic valves and the presence of coronary artery stents. Furthermore, its clinical usefulness is limited.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    The idea of being able to diagnose cases of vasculature without the use of physiological contrast has always had great appeal. It is not only eliminating the injection of chemical agents inside the human body but being able to avoid it would mean great savings for the health system, in economic terms. The emergence of new image segmentation techniques based on convolutional networks has made it possible to address previously intractable challenges. The segmentation of non-contrast cases is one of them. This article is a further step in the process of replacing harmful imaging techniques with others, using the computer. Although it is not to be expected that the need for physiological contrast injection in the patient will be eliminated immediately, it should be noted that articles such as this one will help to do so. Therefore, although for the time being cases including images with physiological contrast will continue to be used in radiology departments, it is good to develop techniques that will allow its progressive elimination.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a method for aortic valve segmentation on CT images. The paper includes cases from different manufacturers, although most of them come from Siemens CTs. The result obtained is particularly striking, given that the aim is to obtain this volume in non-contrast images. This fact makes the proposed method particularly challenging from a technical point of view. However, from a clinical perspective, it is more difficult to justify, since contrast-enhanced images are usually used. Even so, it seems interesting to be able to advance in this type of techniques that avoid the injection of physiological contrast.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Accept — should be accepted, independent of rebuttal (5)

  • [Post rebuttal] Please justify your decision

    I stand by my assessment. Certainly, the authors provide the code. It is not available at this moment, but it is understood that it will be available if the paper is accepted. Regarding other comments made by the reviewers, certainly, several aspects of the paper related to metrics and others may be debatable. However, I consider that overall, it is a good and complex work that has been approached with certain guarantees. Among them would be the inclusion of cases from different CT scanners.



Author Feedback

We thank the reviewers for the valuable comments. R#1 Q6 (rejected cases): Bias: This procedure might introduce bias, but evidence suggests otherwise. Registering contrast and non-contrast CT images is challenging due to their differences, with success varying by registration and case specifics. We hypothesize that poor registrations are not linked to the anatomical properties of the aortic root. By rejecting cases with incorrectly registered ostia, we believe our dataset is improved; Scalability: Rejecting poorly registered cases is efficient, taking <1 min/case, much faster than delineation or manual 3D modeling, making it scalable; Performance: The model’s precise reconstruction of the aortic root suggests no significant bias. We compared two meshes with anatomical information from contrast and non-contrast CTs. Rigid registration accurately identified corresponding regions, supporting our approach’s validity. R#1 Q6+Q10 (other metrics): Indeed, providing more metrics could be beneficial. However, we believe our measure is appropriate for the following reasons: Suitability of Dice: The aortic valve (AV) in our modeling is a 2D surface object, and Dice is best for comparing volumes. Analyzing a section of the aortic root with the AV would result in varying outcomes based on the section, so we opted for surface-based measures; Analysis of Distance Distribution: We analyzed the distance between the non-contrast AV mesh and the registered contrast AV mesh. The std. dev. averaged around 0.6mm. By estimating Hausdorff-95 as the mean distance plus 2 std. dev., we get approx. 2mm. Fig. 3 shows this distance with a colormap on the registered mesh (range: [-2mm, 2mm]); Advantages of Mean Distance: The mean distance is less sensitive to insignificant AV details not fully resolved in non-contrast CT. Significant deviations were found in the middle of the AV leaflets, parts of the ostia, and leaflet positions (relevant to this study). Thus, we used the mean distance, which is less affected by outliers. The non-contrast 1.5mm z-spacing provides an absolute scale for our findings. R#3 Q6.1 & R#1 Q6 (novelty): Our novelty lies in combining SOTA, open-source codes to address novel problems, ensuring high reproducibility and focusing on real-world applications. By tackling unresolved & challenging problems in a non-obvious manner while leveraging SOTA, we believe our contribution is valid and impactful. R#3 Q6.2 (dataset): While AV segmentation in non-contrast CT is typically independent from AV calcification detection, it is crucial for identifying AV calcium, as in [7], which drives our work. Our models were also trained on scans with AV calcifications. Although we showed results on an open dataset, we evaluated the method on scans with AV calcifications and found no influence on AV segmentation. R#3 Q6.3 (evaluation & limited AV visibility): A key result of our work is the quantitative evaluation method for AV reconstructions in our ML model. Using our ICP-based procedure, we identified local transformations of the AV region by utilizing the visible part of the aortic root. This approach allows for an exact comparison of the AV, which is not directly visible in CT images, as shown in Fig 3. It is one of the most comprehensive approaches available, as direct segmentation of the AV in non-contrast CT is not feasible. R#3 Q6.4 (comparison with SOTA): Directly comparing with SOTA methods is challenging due to their limited reproducibility. Based on visual inspection, our approach demonstrates superiority by accurately segmenting the AV, unlike other methods that yield coarse segmentations of the aortic root. R#3 Q8 & R#4 Q8 (source code): As stated on page 7, we provide access to the code for the ICP Method for Accuracy Estimation. The other methods are based on open components. R#3 Q10 (errors): Thank you for noticing the error in Fig. 3, we’ll correct it. R#4 Q6 (anomalies): Thanks for this comment. We will include it in future papers.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    Accepts

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    Accepts



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The paper presents a simple idea that works well. There might not be novelty in the individual components per se, but they are combined in an effective way. This approach could also have value for segmentation of other non-contrast-enhanced CT images. The paper is well-prepared.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The paper presents a simple idea that works well. There might not be novelty in the individual components per se, but they are combined in an effective way. This approach could also have value for segmentation of other non-contrast-enhanced CT images. The paper is well-prepared.



back to top