Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Achieving accurate vessel segmentation in medical images is crucial for various clinical applications, but current methods often struggle to balance topological consistency (preserving vessel network structure) with segmentation accuracy (overlap with ground-truth). Although various strategies have been proposed to address this challenge, they typically necessitate significant modifications to network architecture, more annotations, or entail prohibitive computational costs, providing only partial topological improvements. The clDice loss was recently proposed as an elegant and efficient alternative to preserve topology in tubular structure segmentation. However, segmentation accuracy is penalized and it lacks robustness to noisy annotations, mirroring the limitations of the conventional Dice loss. This work introduces the centerline-Cross Entropy (clCE) loss function, a novel approach which capitalizes on the robustness of Cross-Entropy loss and the topological focus of centerline-Dice loss, promoting optimal vessel overlap while maintaining faithful network structure. Extensive evaluations on diverse publicly available datasets (2D/3D, retinal/coronary) demonstrate clCE’s effectiveness. Compared to existing losses, clCE achieves superior overlap with ground truth while simultaneously improving vascular connectivity. This paves the way for more accurate and clinically relevant vessel segmentation, particularly in complex 3D scenarios. We share an implementation of the clCE loss function in https://github.com/cesaracebes/centerline_CE.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/1081_paper.pdf

SharedIt Link: https://rdcu.be/dZxew

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72111-3_67

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/1081_supp.pdf

Link to the Code Repository

https://github.com/cesaracebes/centerline_CE

Link to the Dataset(s)

https://www5.cs.fau.de/research/data/fundus-images/ https://personalpages.manchester.ac.uk/staff/niall.p.mcloughlin/ https://zenodo.org/records/4521044 https://figshare.com/articles/figure/FIVES_A_Fundus_Image_Dataset_for_AI-based_Vessel_Segmentation/19688169 https://asoca.grand-challenge.org/

BibTex

@InProceedings{Ace_The_MICCAI2024,
        author = { Acebes, Cesar and Moustafa, Abdel Hakim and Camara, Oscar and Galdran, Adrian},
        title = { { The Centerline-Cross Entropy Loss for Vessel-Like Structure Segmentation: Better Topology Consistency Without Sacrificing Accuracy } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15008},
        month = {October},
        page = {710 -- 720}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper presents a centerline-Cross Entropy (clCE) loss for vessel-like structure segmentation. The values of clCE are compared to the values of centerline-Dice (clDice) loss for method evaluation.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The authors explored the efficacy of cross entropy loss and assessed that against the dice loss.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The novelty and technique contribution is insufficient. The model training follows pretty standard procedures. The authors replace the dice loss by cross entropy loss that has been widely used in literature. There seems to be errors in the key equations (5) and (6), see comments for authors below. The validations and results are insufficient as well. Only loss values are compared, which do not indicate the effectiveness or accuracy of the method. Thus this study is far from addressing the challenge of achieving accurate vessel segmentation while preserving topological consistency as stated by the authors. Furthermore, the future work of exploring advanced techniques for skeleton representation and graph-neural networks to further enhance segmentation performance has already been done, see for example

    A comprehensive survey on segmentation techniques for retinal vessel segmentation, Neurocomputing 556 (2023) 126626, https://doi.org/10.1016/j.neucom.2023.126626. State‐of‐the‐art retinal vessel segmentation with minimalistic models, Scientific Reports (2022) 12:6174, https://doi.org/10.1038/s41598-022-09675-y A Detailed Systematic Review on Retinal Image Segmentation Methods (2022), https://doi.org/10.1007/s10278-022-00640-9.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Patches of images of 512x512 were extracted for training. Training parameters and procedure details were not stated.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Numerous studies have been published on vessel segmentations using deep learning with various network architecture designs and loss functions. See some articles listed above. The performance of the method should be evaluated against other methods on metrics such as sensitivity, specificity, accuracy, AUC, FPR, MCC, etc. Ablation study would also be desired. There seems to be errors in equations (5) and (6). They both miss a negative sign for CE loss. The summation in eqn (5) looks incomplete. In eqn (6), should the P be P_hat and the S_T be S_{P_hat} in row 1? And should the third T_i inside log be P_hat_i in row 2? The summation range for i needs to be stated.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Strong Reject — must be rejected due to major flaws (1)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper lacks the novelty/originality and technique contributions. The method validations are insufficient and limited. The equations contain errors. Training parameters and procedure details are not provided.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Review #2

  • Please describe the contribution of the paper

    This paper introduced the centreline-cross-entropy (clCE) loss for segmentation of tubular structures. The clCE loss reformulates the existing clDice loss using cross-entropy, improving robustness to noise and without sacrificing segmentation accuracy. The method is tested on numerous retinal imaging datasets and shown to improve the Dice coefficient compared to the standard Dice loss or clDice loss.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The paper is well written and easy to follow. The review of the Dice coefficient and loss is simple but helpful, when then sets the stage for the review of the clDice loss and the introduction of the proposed clCE loss. The main challenge is to recast the binary clDice loss using non-binary images, which is achieved using the soft-skeletonization method of [27] which then feeds into Equations 6 and 7. Overall, the formulation is simple and elegant. The code on Page 6 is simple to implement and due to its simplicity, it becomes easy to use this method for segmentation problems.

    Figure 1 is helpful in showing practically, the effect of the clCE loss and how it may improve a segmentation compared to the clDice loss.

    Experiments are performed using a variety of retinal image datasets and a CT cardiology dataset. Inclusion of pathological cases is helpful to show how the method may handle challenging examples. Results show training with Dice + clCE loss produces a better Dice score than using the Dice loss or a combination of Dice + clDice loss. Unsurprisingly, the method may underperform on the cl-DSC metric compared to using the cl-DSC loss.

    Source code is planned to be released, this is encouraged and will make it easier for others to reproduce the results and implement the loss in their own work.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Although paper has many strengths mentioned above, it also has weaknesses.

    First, while the paper provides compelling numeric results in Tables 1, 2, and in the supplementary material, it would be more compelling and convincing to have an additional figure with visual results. Figure 1 is helpful to show an example, but the reader is left wondering if it is a representative example? Qualitatively, how does the method perform on the retinal images?

    Relatedly, the paper evaluates the performance using DSC and cl-DSC. One would expect clDice to perform better on the cl-DSC metric as clDice is designed to optimize this metric. Unlike clDice which trades off DSC or cl-DSC, the proposed method improves both DSC and cl-DSC. Something confusing in Table 1 is how the increases/decreases in green/red are computed. For example, the 5th column as DSC of 79.34, which is an increase of +6.37, but it wasn’t clear how +6.37 was derived, shouldn’t it be (79.34-78.93)= +0.38? Please check the tables for accuracy and correctness. This reviewer was curious regarding Equation 4, does the proposed method perform better on topological precision or recall?

    Overall, the improvements with the proposed method are somewhat modest. Nonetheless they look meaningful. Again it’d be helpful to see qualitatively how the method performs as well to provide additional evidence the method is truly effective.

    One Page 2, the paper talks about a strategy to build loss functions than enforce topological consistency. However, the proposed method does not provide any guarantees of topological correctness. How to enforce topological correctness? Maybe the paper means “encourage” rather than “enforce”?

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Pseudocode in the paper makes the method easy to understand and reproduce. Source code to be released.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    Please see the strengths and weaknesses section which provides key constructive comments.

    Small issue • Please consider capitalizing each word in the title (except small word like “for”

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Overall, this paper presents a simple but meaningful extension to the clDice loss, producing a new clCE loss which has advantages over clDice. Although the results presentation could be improved, the method appears to improve topological consistency without sacrificing Dice score. The simplicity of the method is an advantage.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Accept — should be accepted, independent of rebuttal (5)

  • [Post rebuttal] Please justify your decision

    Thank you for the rebuttal which clarified some issues raised at the initial review stage. I would like to see this paper in the conference proceedings as it provides a simple way to encourage topological consistency and appears to be an advance for vessel-like structure segmentation.



Review #3

  • Please describe the contribution of the paper

    Accurate vessel segmentation is a challenging task, as topology is not easily accounted for in segmentation loss functions. Voxels belonging to small vessels are more topologically relevant than voxels on the surface of large vessels. Further, accuracy of the segmentation at a voxel level is hardly obtained in reference benchmarks. Loss function engineering for vessel segmentation is an active area of research. The authors present a new loss function: centerline-Cross Entropy (clCE), to promote vessel overlap and maintain network’s structure. They evaluate the method on publicly available 2D and 3D datasets, obtaining better dice coefficient and vascular connectivity to the reference standard.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Simplicity. The experimentation is clear and well thought.

    Extremely clear on the reasoning.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    Adding LclCE to LDice improves results both in the sense of DSC and cl-DSC. After reading the paper one wonders how the combination of LDice + LclDice + LclCe would perform. The combination of the centerline-based loss functions may outperform each of them individually, the same way as LDice + LCE outperforms each of the components.

    Improvements can be made to the equations. • Please define the symbol of the dot with the circle on eq (5) • In the unnumbered equations between Eq. 4 and Eq. 5, please define the doubled-lined P. Is it a probability? • Eq. 6 seems erroneous. Unless error on my end, the second log should be log(1-Pi) instead of log(1-Ti).

  • Please rate the clarity and organization of this paper

    Excellent

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Databases are publicly available. The code will be shared in github.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    I suggest reviewing the equations and performing the experiments mentioned in Weaknesses.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Accept — should be accepted, independent of rebuttal (5)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Good paper with a clear take home message.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    N/A

  • [Post rebuttal] Please justify your decision

    N/A



Author Feedback

We appreciate the provided feedback, and acknowledge the presence of notation errors in our submission (these did not affect the correctness of our formulation and have been fixed now, see below). We also regret having to dedicate a large portion of our rebuttal to responding to a series of unsubstantiated, subjective, and occasionally incorrect criticisms from R4. We attempt to address as many valid improvement suggestions as possible in the second part of this letter.

R4: “The performance of the method should be evaluated on metrics such as sensitivity, specificity, accuracy, AUC, FPR, MCC”, “Only loss values are compared, which do not indicate the effectiveness or accuracy of the method”: R4 is wrong here. The metrics advised by R4 are invalid for assessing topological correctness and are unsuitable in imbalanced scenarios like vessel segmentation. The community consensus is to use DSC in conjunction with cl-DSC; please see pg. 198 in Maier-Hein et al., Metrics Reloaded, Nature Methods 2024. Also, we do not compare loss values, we report DSC and cl-DSC metric values.

R4: “The model training follows pretty standard procedures”: Of course it does. This is a strength of our work, not a weakness. We propose a new loss function, not a new training method. We adopt well-established models and training procedures, like the nnUNet, and deliberately refrain from modifying them. Expecting authors to introduce novel training procedures for the sake of novelty alone is bad reviewing practice.

R4: “The novelty and technique contribution is insufficient”: Subjective opinion with no references to support it.

R4: “The authors replace the dice loss by CE loss that has been widely used in literature”: R4 is wrong. We do not “replace the dice loss by the CE”, but improve topological consistency drawing inspiration from the clDice loss and addressing its weaknesses with a more robust formulation based on the CE loss over the skeletons of the target and prediction.

R4: “Training parameters and procedure details are not provided”: The nnUNet framework automatically adjusts the training procedure. Our retinal vessel segmentation model follows the approach in “State‐of‐the‐art retinal vessel segmentation with minimalistic models”, with minor modifications. As noted in our paper, “Exact training details are available at Github”. It is impractical to include routine training details and hyperparameter values on the paper due to the page limit, this is the purpose of code repositories.

General comments - Notation and equations: Following R1 and R4’s feedback (thank you), we have revised our notation. Errors in equations were related to a last-minute change of notation, but the text was correct and has not been modified. Importantly, the code snippet was also correct. The necessary corrections were:

  • \mathbb{P} meant probability, circled dot was voxel-wise product, this has been made explicit on the text.
  • Eqs. (5) & (6): minus sign added to CE terms.
  • Eq. (6): log(1-Ti) replaced by log(1-Pi), all missing hats on Pi added, summation range explicited.

R1:

  • We are not allowed to add new experiments in rebuttal; however, we tried combining LDice+LclDice+LclCE when developing our method and results were only marginally different.

R3:

  • Qualitative examples: unfortunately, the page limit does not allow us to add these to the paper, we have uploaded all test set segmentations to our Github repo along with the code to reproduce our experiments, and some visual examples on the landing page.

  • Table 1: the error pointed out was a typo when transferring results from a Jupyter notebook to LaTeX. We have reviewed all our numbers and found no additional errors, thanks a lot for spotting this one. We have added further clarification on how to interpret results in the caption for the reader’s convenience. The term “enforce” has been replaced by “encourage” topological correctness.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    The reviewers have expressed differing opinions on this paper. After reviewing both the manuscript and the rebuttal, I am inclined to side with the majority of the reviewers and accept the rebuttal. The proposed centerline-Cross Entropy (clCE) metric demonstrates its utility in enhancing vessel overlap and preserving the network’s structure. The results also support the stated hypothesis. However, the errors in the equations, as highlighted by the reviewers, must be corrected prior to publication. In addition, the format of the tables is unusual and the source code is not suggested to be directly put in the manuscript

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    The reviewers have expressed differing opinions on this paper. After reviewing both the manuscript and the rebuttal, I am inclined to side with the majority of the reviewers and accept the rebuttal. The proposed centerline-Cross Entropy (clCE) metric demonstrates its utility in enhancing vessel overlap and preserving the network’s structure. The results also support the stated hypothesis. However, the errors in the equations, as highlighted by the reviewers, must be corrected prior to publication. In addition, the format of the tables is unusual and the source code is not suggested to be directly put in the manuscript



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    Two reviewers are very enthusiastic about the work and recommend acceptance. One reviewer recommends strong reject, however, the rebuttal clearly identifies and corrects some misunderstandings and flaws in this review.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    Two reviewers are very enthusiastic about the work and recommend acceptance. One reviewer recommends strong reject, however, the rebuttal clearly identifies and corrects some misunderstandings and flaws in this review.



back to top