Paper Info Reviews Author Feedback Meta-Review Back to top
List of Papers Browse by Subject Areas Author List

Abstract

Reconstructing 3D volumes from optical microscopic images is useful in important areas such as cellular analysis, cancer research, and drug development. However, existing techniques either require specialized hardware or extensive sample preprocessing. Recently, Yamaguchi et al proposed to solve this problem by just using a single stack of optical microscopic images with different focus settings and reconstructing a voxel-based representation of the observation using the classical iterative optimization method. Inspired by this result, this work aims to explore this method further using new state-of-the-art optimization techniques such as Deep Image Prior (DIP). Our analysis showcases the superiority of this approach over Yamaguchi et al in reconstruction quality, hard metrics, and robustness to noise on the synthetic data. Finally, we also demonstrate the effectiveness of our approach on real data, producing excellent reconstruction quality. Code available at: https://github.com/caiocj1/multifocus-3d-reconstruction.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2024/paper/0551_paper.pdf

SharedIt Link: https://rdcu.be/dY6id

SpringerLink (DOI): https://doi.org/10.1007/978-3-031-72083-3_5

Supplementary Material: https://papers.miccai.org/miccai-2024/supp/0551_supp.pdf

Link to the Code Repository

https://github.com/caiocj1/multifocus-3d-reconstruction

Link to the Dataset(s)

N/A

BibTex

@InProceedings{Aze_Deep_MICCAI2024,
        author = { Azevedo, Caio and Santra, Sanchayan and Kumawat, Sudhakar and Nagahara, Hajime and Morooka, Ken'ichi},
        title = { { Deep Volume Reconstruction from Multi-focus Microscopic Images } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024},
        year = {2024},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15004},
        month = {October},
        page = {47 -- 57}
}

Reviews

Review #1

  • Please describe the contribution of the paper

    This paper explores using the Deep Image Prior technique to improve 3D reconstruction quality and robustness compared to previous iterative optimization method.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The analysis aims to demonstrate the performance of 3D reconstructions on synthetic and real data.
    • Great suggestion on pretraining of the DIP Network.
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    • The main focus of the paper is to use a 3D U-Net to perform the reconstruction instead of the iterative optimization way. Figure 1 is misleading, as the expected usefulness of the model should take 2D multi-focus images as input to do the reconstruction. Figure 1 only shows the input of 3D random noise, which is very misleading.

    • Synthetic data may not accurately capture the complexity and variability of real-world data, leading to models that perform well on synthetic data but poorly on actual data. Additionally, the process of generating synthetic data might introduce biases if the underlying assumptions or methods used are not representative of real-world conditions.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html

    • The figure 1 should indicate that the right-hand side is the baseline method.

    • Some of the results are very close, so statistical differences should be provided.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Clarification of model’s usefulness. Statistical Differences. Synthetic vs. Real-world Data.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Reject — could be rejected, dependent on rebuttal (3)

  • [Post rebuttal] Please justify your decision

    Thank you for your detailed response. While I understand that DIP is trained on a per-sample basis and does not require pre-training, it is still important for the synthetic data used in demonstrations to accurately reflect the complexity and variability of real-world data. Any biases in the synthetic data generation process can affect the model’s performance. Therefore, validating the method on real-world datasets is essential to confirm its robustness and practical applicability. Additionally, I noticed that the performance statistical analysis was not addressed in the response. Given these points, I will maintain my previous score.



Review #2

  • Please describe the contribution of the paper

    This paper discussed the problem of reconstructing volumetric data from multiple micripscopic images. The authors propose to use multi-focus bright field microscripic samples to synsthsive data and producing good quality reconstruction data. Based on the iterative optimization method, from Yamaguchi et al’s work, this work adapted Deep Image Prior to improve the volume reconstruction.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The authors target a significant task on reconstructing or generate volume from microscopic images. The application has potention value for many clinical problems.

  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

    The background introduction is generally well structured. However, it’s hard to follow what is the challenge of the “classical iterated methods”. The authors can discussion more on the root metrics or observation on why deep image prior is needed based on the state-of-the-arts. What does propose a benchmark datasets from the Open-Scivis” mean? Does the authors released the initial dataset or re-compose the dataset? Does this claimed as one of innovations? To me, it’s sounds like this work cherry picked the samples from an existing dataset and claimed improvements. The method illustrated in this paper, used an 3D UNet as reconstruction backbone, , generate multiple reconstructed images and calculated L1 loss and optimize the model. Technically, the authors can discussion why generative models are not used or compared in this task, such as GAN-based, diffusion-based model that are dominating generative image tasks. The experiment design and technical setting lacks basic comparisons and discussion of generative AI baselines.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    Upon reproducibility, recommend to comment on the potential code release or accessibility.

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    1. The authors further discussion the challenges in terms of technical. the problem and clinical value is structured well. But the this paper lacks comprehensive technical discussion.
    2. to improve technical merit, the authors can compare and conduct experiments on the generative AI researches.
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Though this paper lacks a technical merit, the authors focus on a significant problem and innovative way of learning microscopic images.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    The authors rebuttal response are good and clearly explained comments and concerns.



Review #3

  • Please describe the contribution of the paper

    A new reconstruction method for 2D to 3D microscopy using the deep image prior framework.

  • Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • New reconstruction method improves image quality
  • Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
    • A more thorogh validation of the reconstructions would be great. E.g. couldnt you use fluorescent images to test object locations and properties that your approach predicts deep in the tissue
    • Code is missing to use and test method
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Do you have any additional comments regarding the paper’s reproducibility?

    N/A

  • Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review. Pay specific attention to the different assessment criteria for the different paper categories (MIC, CAI, Clinical Translation of Methodology, Health Equity): https://conferences.miccai.org/2024/en/REVIEWER-GUIDELINES.html
    • A more thorogh validation of the reconstructions would be great. E.g. couldnt you use fluorescent images to test object locations and properties that your approach predicts deep in the tissue
    • some typos should be remedied
  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Method seems to work, but I would love to see more thorough, more challenging evaluations.

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the author’s rebuttal, state your overall opinion of the paper if it has been changed

    Weak Accept — could be accepted, dependent on rebuttal (4)

  • [Post rebuttal] Please justify your decision

    Thanks for addressing the reviewers’ issues. Please (1) adapt the final version to explain things better, and (2) reduce the number of digits you show in your tables to the significant ones. Evaluate mean and s.d. by iterate experiments.



Author Feedback

R1, R3, R4: Thank you very much for the useful comments that have helped us to further improve our work.

[Q1] Code and Reproducibility. [A1] Our apologies for not releasing the code during initial submission. Unfortunately, the rebuttal rules now forbid adding new files or links. However, we will release a GitHub link in the final version.

R1: [Q1] Clarification of model’s usefulness. [A1] Kindly note that our method is not an image transformation approach such as regular image-to-image transfer or images-to-volume transfer by using U-Net. DIP using 3D U-Net is like a generative approach, directly generating the 3D target volume from the noise. Then the observed images are generated from the estimated 3D volume and the loss is calculated between the simulated images and observed images. The loss is back-propagated to train the DIP model iteratively. Hence, figure 1 correctly shows this pipeline.

[Q2] Synthetic vs. Real-world Data. [A2] Please note that DIP is a generative model that is trained for sample by sample. The model is not pre-trained such as the image-to-image transformation model by regular U-Net. Hence there is no question of domain gap between the synthetic and real data.

R3: [Q1] Classical iterative vs DIP method. [A1] Please note that we do not claim the “classical iterative” method as our proposal. It is just an example of the classical explicit regularization model, TV prior, to be compared with the characteristics of the deep learning-based prior, DIP, in our proposed methods.

[Q2] Open-Scivis dataset clarification. [A2] Kindly, note that, to the best of our knowledge, there is no dataset for non-linear observation images with a bright field imaging model with ground truth voxel attenuations. Hence, we use Open-Scivis as the ground truth and assume arbitrary attenuations on the 3D shape, then apply our observation model to simulate the imaging of multi-focus images captured by bright field microscopy. The Open-Scivis dataset consists of a total 55 samples and we randomly sampled 13 samples for all our experiments.

[Q3] Discussion and comparison with generative AI baselines. [A3] Thank you pointing this out. Please note that one of the major challenge of our work is that it deals with translucent microscopic objects. Unfortunately, we are not aware of any GAN / diffusion-based methods that deal with such objects, as they mostly deal with opaque objects. However, we do plan to identify and modify existing generative AI methods for our setting and compare with them in an extended work in future.

R4: [Q1] More thorough validation of the reconstructions using fluorescent images? [A1] Thank you for the suggestion. Kindly, note that, we can also apply DIP for fluorescent images. However, the main contribution of our paper is applying DIP to attenuation volume reconstruction from the bright field images.

The imaging model for fluorescent images is modeled by the linear convolution in which the voxels themselves emit fluorescence and the rays from the voxels form a blur of the captured images based on different focal settings. Note that most DIP-based papers deal with linear observation or degradation models, such as deblurring, denoising, and super-resolution in 2D and 3D reconstruction.

On the other hand, our attenuation voxel reconstruction from bright field microscopy is similar to CT reconstruction in which the light ray comes from outside of the object, and the rays are attenuated by the voxels along the ray direction. It is non-linear as modeled by Equations 1 and 2 in the main paper. Hence, our contribution is to show that DIP is applicable to such an non-linear imaging model also.

[Q2] About code and typos. [A2] We will address these issues in the final version.



Meta-Review

Meta-review #1

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    This application study presents a method to recover 3D geometry from translucent volumetric images. A clear contribution is showed on public benchmark datasets using the previously published Deep Image Prior approach. There is lot of work in the automatic transfer function design domain for direct volume rendering that can also be discussed in the final version.

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    This application study presents a method to recover 3D geometry from translucent volumetric images. A clear contribution is showed on public benchmark datasets using the previously published Deep Image Prior approach. There is lot of work in the automatic transfer function design domain for direct volume rendering that can also be discussed in the final version.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation. You may optionally write justifications for ‘accepts’, but are expected to write a justification for ‘rejects’

    N/A

  • What is the rank of this paper among all your rebuttal papers? Use a number between 1/n (best paper in your stack) and n/n (worst paper in your stack of n papers). If this paper is among the bottom 30% of your stack, feel free to use NR (not ranked).

    N/A



back to top