Abstract

Lack of adequate ground truth data is a common issue in various medical image processing tasks. Especially in non-uniform image registration and image reconstruction, dense manual labeling of data is both infeasible and insufficient in terms of accuracy, which prevents the use of a manually-labeled test set. In contrast, simulation can offer unlimited data with exact ground truth. However, since simulations are based on models that are limited themselves, the evaluated algorithm could be overfitted to the limited simulation model, and therefore perform worse in real data (the “realism gap”). In this paper, we use combinations of simulated and real data to quantitatively reveal the gap: additional degrees of variation in real data worsen the compensation of the simulated data, introducing a mismatch with its ground truth. Next, we present meta-ablation, a concept to match the simulation to the real data. Inspired by ablation studies, this is achieved by joint variation of the simulation and the compensation method while observing and ideally closing the mismatch. Our approach is intended for fusion tasks that compensate for confounders in repeated measurements. Examples include motion and distortion compensation, group-wise registration, and multi-frame super resolution. We demonstrate our approach in the evaluation of non-uniform image registration of 3D OCT data. We not only show that model advances allowed closing the realism gap, but also that inspecting residual errors from limited models in the combined setting allowed us to sequentially deduce the most important next model advance. In short: Our (in real data) label-free approach validates simulated data and can, in certain applications, provide a strategy for self-optimization.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5527_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

N/A

Link to the Dataset(s)

N/A

BibTex

@InProceedings{PloSte_Validation_MICCAI2026,
        author = { Ploner, Stefan B. AND Schneider, Linda-Sophie AND Hwang, Yunchan AND Hassan, Abdul R. AND Waheed, Nadia K. AND Fujimoto, James G. AND Maier, Andreas},
        title = { { Validation of simulation-based evaluation by inclusion of unlabeled data and meta-ablation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16881},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper introduces meta-ablation, a framework that quantifies sufficiency of simulated data by comparing method performance on fully simulated versus mixed simulated-and-real data. The resulting “realism gap” serves as a label-free diagnostic for simulation insufficiency, while the outcomes guide targeted model improvement.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The proposed framework is elegantly simple and practical. It requires no additional labels, external references, or specialized tools, but relies just on the comparison of performance on all-simulated versus mixed data within an existing evaluation pipeline.

    • The error analysis directly reveals which physical effects are missing from the simulation, which helps guide model improvement. Thereby, the framework enables label-free estimation of real data properties.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The evaluation relies primarily on the L2-norm. While the authors demonstrate that per-component evaluation can reveal gaps hidden by the aggregate norm, the inclusion of fundamentally different metrics would have strengthened the claim that a closed realism gap reflects genuine simulation realism rather than insensitivity of a particular metric.

    • The paper only tests a single ratio of simulated and real data. Even though a larger ratio of real data would limit data for quantification, varying this ratio would provide insight into the severity of the gap, since a single real volume causing large changes would indicate a fundamentally different problem than a gradual shift proportional to the amount of real data introduced.

    • In the results, the authors state “In the right part, the same models are evaluated”, but a reference to the corresponding figure is missing (probably Figure 3) and the right part of Figure 3 does not contain the same models as it is cut off at Pinoeer22.Showing the results for the other models would be interesting. Additionally, it is stated “This realism gap is then also closed with the Pioner25 model. ”, again lacking reference to the corresponding figure (maybe Figure 4, left?). The same issue continues for the following sentences, which makes traceability of the argumentation difficult.

    • The effectiveness of the framework depends on the method using an explicit, physics-based parametric model where identified missing effects can be directly added as new parameters. This coupling may break down for deep learning-based approaches. A CNN-based method may benefit from the outcome that a realism gap exists, but offers no clear path from outcome to model improvement, since the network’s internal representation typically cannot be directly augmented with specific physical parameters. The authors do not discuss the applicability of their framework to learning-based methods, which limits its relevance to an increasingly large share of the field.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper presents a novel, well-motivated framework addressing a real gap in the validation of fusion-based tasks. The core idea is simple and applicable for physics-based methods. However, limited experimental extend (single metric, single data ratio, single site) and inapplicability to deep learning-based methods constrain its impact, making it a solid conceptual contribution with room for stronger validation.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper proposes “meta-ablation”, a label-free framework to stress-test whether a simulation is adequate for evaluating an image-fusion method when real-data ground truth is unavailable. The core idea is to compare two settings: evaluation on fully simulated fused inputs, and evaluation on fused inputs where part of the group is replaced by real unlabeled data. The performance drop on the simulated part is interpreted as a realism gap: by repeating this under different simulator/model assumptions and inspecting residual error maps, the authors aim to identify missing model components. The paper demonstrates this on retinal OCT motion correction.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The addressed problem (assessment of a potential realism gap) is relevant for the MICCAI community.

    • While the method is not applicable to every problem (mostly for fusion-based tasks), it is still very generic.

    • The proposed approach is elegant and simple, and seems easy to try out despite the lack of source code.

    • The experiments are interesting, especially Figure 3 which compares different kinds of simulated data and whether a realism gap can be observed or not.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • The paper is not really self-contained and it was a bit hard to understand the application for people who don’t have experience in OCT motion compensation. Many information and implementation details are deferred to prior OCT papers. I would recommend to go over Section 1 and 2 reformulate some sentences.

    • As the method is explicitly formulated in very generic terms, it would have been even more convincing to try it on two different applications. Maybe this would also made the methods description even clearer.

    • I am not sure I agree with the claim that the method proves “sufficiency”. I understand that a realism gap can be exhibited using that method, but the absence of a difference doesn’t mean that there is no realism gap, does it? Maybe some claims should be reformulated.

    • The method is not compared against any other baseline.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
    • I have some doubts on the conclusions drawn by changing both the model and the simulator at the same time. Does the gap close because the simulator is more realistic, or because the model is more flexible, or some entangled behaviour of the two? Would it make sense to add experiments where only the simulator is changed to rule out this hypothesis?

    • Why was a sign test chosen? A more standard test (like Wilcoxon signed rank) would be better suited, since the error are numeric values, not binary ones.

    • I am not sure to understand why the evaluation of downstream tasks is so quickly discarded. In what sense are they too “abstract”?

    Typos and language improvement:

    “insufficfient” → insufficient “It’s importance” → Its importance “objected detection” → object detection “loose details or accuracy” → lose details or accuracy “reasearch” → research “limiations” → limitations

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I am leaning towards acceptance because, even if the study has limitations, the method is simple and can give ideas to the community beyond the OCT application presented here. I would be interested to know in the rebuttal whether the authors agree that some of the claims should be toned down, or whether I misunderstood some of the experiments (in which case I can upgrade my rating).

  • Reviewer confidence

    Somewhat confident (2)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper addresses the fundamental problem of how to demonstrate that simulation-based test data is sufficiently realistic when non ground truth exists for real data. The proposed method of meta-ablation extends the classic ablation study paradigm to vary simulation complexity while including real (unlabeled) data.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    S1 : The meta-ablation idea is a genuine contribution to evaluation methodology.

    S2 : Sign test with p<0.001 and 90 aggregate values per condition.

    S3 : Limitations clearly layed out, good scientific rigor.

    S4 : Not requiring to label real data is a strong practical advantage.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    W1: Non scientifically supported claims of applicability to registration, reconstruction and super-resolution but only demonstrates for one OCT registration task.

    W2 : Single task, single modality, specific subpopulation : the demonstration uses only retinal OCT data, from 15 patients subject to the same pathology.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    Q1 : Your framework assumes the gap manifests as consistent degradation

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Strong methodological paper. The meta-ablation concept is orginal, the demonstration is rigorous, and the paper is honest in its listing of limitations. While the single-task, single modality demonstration limits the claim of large scale applicability, the core contribution is sound.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

We thank the reviewers and meta-reviewer for their valuable comments, the appreciation of the novelty in our approach, and the positive evaluation. As mentioned in the following, we did evaluate several more subaspects, but as the manuscript is limited to 8 LNCS pages, we needed to restrict ourselves to the most insightful ones for this paper. We plan to include the more comprehensive evaluations in a future journal article.

EXTENSION REQUESTS

R1: “The evaluation relies primarily on the L2-norm. […] the inclusion of fundamentally different metrics would have strengthened the claim that a closed realism gap reflects genuine simulation realism rather than insensitivity of a particular metric.”

  • We typically compute errors for 3D euclidean error (L2-norm of the displacement vector), error on x,y,z axes individually, and error along the fast and slow scan directions. This allows closer investigation which axes cause the realism gap & where additional parameters are needed. For the purpose of brevity and clarity, we only reported the combined, L2 error in Fig. 3.(Fig. 4 shows z- and y-specific errors instead, which allow to separate the missing effects of axial curvature and transverse scale compensation.)
  • Regarding the metric, we totally agree that multiple metrics should be used. Aside from the median error, we also computed the percentage of position errors above a threshold. Although not included to meet the page limit, the results were consistent. We will add a brief note of this consistency to the manuscript.

R1: “The paper only tests a single ratio of simulated and real data. […]” We reported on as much real data as possible (setup 1 simulated, 5 real) to maximize overall realism and to maximize the size of realism gaps. Limited experiments with fewer real volumes showed reduced realism gap sizes, with a stronger increase when including the first real dataset. We also investigated specific subsets of simulated volumes (all parallel / all orthogonal scans real), but could not derive meaningful insights -> omitted.

R2: “[…] it would have been even more convincing to try it on two different applications.” R3 W2: “Single task, single modality, specific subpopulation” Due to the page limit, we opted to demonstrate the potential for model improvement (Fig. 4) as an additional advantage of the method, instead of a second application.

OTHER

R1: does not apply to learned approaches We agree that the model-understanding extension (Fig. 4) naturally only applies to physics-based models, but want to stress that the core meta-ablation idea does apply to learned approaches.

R2: “I am not sure I agree with the claim that the method proves ‘sufficiency’.” We agree with the reviewer. While this was stated very clearly in the discussion of the initially submitted manuscript (“allows statistical demonstration of insufficiency […] Although a contrary conclusion of sufficiency is not permissible […]”), we revised formulations in the abstract and outline sections.

R3 (W1): applicability to reconstruction & super-resolution We agree that this is not demonstrated. However, the two core requirements - possibility of simulation, and fusion task that can combine simulated and real data - are given for the mentioned tasks, so we are convinced that our approach could be applied.

R3 (Q1): consistent degradation Already a significant number of degradations, not necessarily consistent degradation in all cases, are sufficient to show statistical effects and differences in the distribution plots, which is what the realism gap / meta ablation is based on.

Finally, I had concerns about the symmetry requirement of the Wilcoxon signed rank test, but we will reevaluate and potentially substitute the significance test for the final submission.




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper proposes meta-ablation, a framework that exploits the difference between real and simulated data to identify potential issues with model performance in the field. I agree with all reviewers that this is a valuable contribution.



back to top