Abstract

Color fundus photography (CFP) is the most common ophthalmic imaging modality for large-scale screening. However, it is highly susceptible to degradations, making robust fundus image quality assessment (FIQA) crucial. The criteria for what constitutes high-quality at the image level vary across clinical tasks, making FIQA dependent on expert knowledge. This motivated the development of automated methods and datasets. While existing datasets aim to standardize image-level quality, their criteria often differ. Furthermore, image-level labels preclude the quantitative evaluation of localized degradations, which is essential for trustworthy FIQA. We argue that pixel-level FIQA based on anatomical visibility represents a more task-agnostic, explainable approach. In this work, we introduce FunPiQ, the first FIQA benchmark to provide pixel-level quality annotations. In addition, we propose EFIQA-CP, an explainable-by-design (EBD) method that uses quality pseudo-labels based on anatomical visibility to train a CNN via Non-Negative Positive-Unlabeled learning. Extensive evaluations of classification methods with post-hoc explanations, anomaly detection methods, and EBD methods demonstrate the superior performance of the last and, particularly, of EFIQA-CP. Our code, weights, and dataset are publicly available at https://github.com/penway/FunPiQ

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/3819_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/penway/FunPiQ

Link to the Dataset(s)

FunPiQ dataset: https://github.com/penway/FunPiQ

BibTex

@InProceedings{WanPen_FunPiQ_MICCAI2026,
        author = { Wang, Pengwei AND Morano, José AND Mares, Virginia AND Bogunović, Hrvoje},
        title = { { FunPiQ: A New Benchmark for Pixel-Level Quality Assessment in Fundus Images } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16895},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The paper introduces FunPiQ, the first benchmark dataset for fundus image quality assessment (FIQA) that provides pixel-level quality annotations based on anatomical visibility. Additionally, it proposes EFIQA-CP, an “Explainable-by-Design” (EBD) framework. This model improves upon the existing EFIQA by incorporating a shallow CNN adapter for better global context and utilizing Non-Negative Positive-Unlabeled (nnPU) learning to mitigate the noise in anatomical pseudo-labels.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    Novel Dataset: The introduction of pixel-level annotations is a significant step forward from traditional image-level labels, addressing the need for fine-grained, task-agnostic quality assessment.

    Methodological Soundness: Transitioning from post-hoc explanations to an “Explainable-by-Design” approach enhances the clinical trustworthiness of the AI system.

    Addressing Label Noise: The use of nnPU learning is a clever and scientifically justified way to handle the inherent noise in pseudo-labels derived from anatomical segmentations.

    Comprehensive Evaluation: The paper compares the proposed method against a wide range of baselines, including anomaly detection and post-hoc explanation methods.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Incomplete Technical Specifications: The manuscript lacks critical training details. There is no mention of the optimizer used, specific learning rates, batch sizes, or data augmentation strategies, which hinders reproducibility.

    Ambiguous Clinical Translation: While the pixel-level Dice and QWK scores show improvement, the paper does not sufficiently discuss how these metrics translate into actual clinical utility or diagnostic reliability.

    Visual Communication Issues: Figure 1, which describes the pipeline and dataset, is visually congested and lacks a clear logical flow. The distinction between data selection, manual annotation, and the supervision process is difficult to follow.

    Unjustified Architecture Choices: The choice of the “shallow CNN adapter” is mentioned as a solution for global context, but there is limited ablation or theoretical justification comparing it to other potential bridging architectures (like ViT-based adapters).

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The introduction of the FunPiQ dataset is a highly valuable contribution to the ophthalmic imaging community. However, the paper in its current form feels unpolished in its technical reporting. The absence of hyperparameter details and the cluttered nature of the primary system figure (Fig 1) make it difficult to fully validate the experimental results. If the authors provide a more rigorous technical appendix and improve the visual clarity of their methodology, this could easily move to an “Accept.”

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper introduces a pixel-level FIQA benchmark dataset with expert annotations FunPiQ, the first dataset specifically designed for pixel-wise quality assessment in fundus images, addressing a significant gap in the field.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.FIQA is a valuable benchmark dataset. The expert-annotated labels and clinically meaningful categorization (good / usable / bad) make this dataset particularly relevant for real-world applications. 2.The experiments include multiple baselines (supervised, explainable, and anomaly detection methods), diverse datasets, and multiple evaluation metrics.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Dataset limitation. The FunPiQ dataset contains only 300 images. While pixel-level annotation is costly, this limited size may compromise the statistical reliability and representativeness of the benchmark. 2.Unfair comparison. The proposed method is trained on a proprietary internal dataset, whereas other methods are trained on public datasets or use pretrained weights. Given that this paper establishes a new benchmark, this setup raises concerns about fairness. 3.Insufficient ablation. In Table 2, the proposed EFIQA-CP and the baseline EFIQA show comparable performance. It remains unclear whether the improvement stems from architectural changes or training data. At a minimum, both models should be trained on the same dataset to fairly demonstrate the superiority of the proposed architecture. 4.Missing quantitative evaluation. One advantage of pixel-level annotation is the ability to quantitatively measure the accuracy of attention maps, rather than relying solely on qualitative visualization. Therefore, in addition to Fig 2, quantitative results should be reported for comparing quality maps.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The tiny FunPiQ dataset compromising benchmark reliability; unfair comparisons; insufficient ablation; and lack of quantitative evaluation for attention maps.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors have addressed all my concerns. I recommend acceptance.



Review #3

  • Please describe the contribution of the paper

    The paper introduces FunPiQ, the first public benchmark providing pixel-level quality annotations for fundus image quality assessment. The dataset comprises 300 images sourced from three existing datasets (EyeQ, BRSET, mBRSET) across two countries, annotated by expert readers under ophthalmologist supervision into the same three classes in pixel-level quality based on anatomical visibility. Besides, the authors propose EFIQA-CP, an improvement over the existing EFIQA method, which replaces the original linear adapter with a wide-yet-shallow CNN featuring a larger receptive field, and substitutes BCE loss with nnPU learning to handle noisy vessel-based pseudo-labels.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.Addresses a genuine gap in previous datasets: Pixel-level quality annotation is a meaningful step beyond entire image labels, as degradations in CFP are inherently local. The decision to base quality criteria on anatomical visibility rather than task-specific clinical requirements is a principled design choice that improves cross-task generalizability, and the multi-source, multi-device, multi-ethnicity sampling adds practical relevance.

    2.Comprehensive and fair benchmark: The paper evaluates seven methods spanning three distinct paradigms on the same held-out benchmark, with consistent statistical testing (large bootstrapping). The use of multiple complementary metrics covering both ordinal agreement and binary rejection tasks provides a thorough and well-rounded comparison.

    3.Targeted and well-justified method improvements: The two modifications in EFIQA-CP (wider receptive field and nnPU) are both directly motivated by identified failure modes of EFIQA (foveal false positives and pseudo-label overfitting). The ablation study confirms the contribution of each component, lending credibility to the design choices.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.Limited Dataset Size and Questionable Sampling Strategy: The dataset contains only 300 images, and while this scale is arguably acceptable given that FunPiQ is intended solely as an evaluation benchmark with costly pixel-level annotation, the sampling strategy requires stronger justification. All images are pre-selected from existing datasets based on having been labeled as “bad” at the image level, meaning the benchmark entirely excludes images of acceptable or good overall quality. This creates a distribution that is unrepresentative of real clinical screening scenarios, where the majority of images are of sufficient quality. The authors do not justify why this selection strategy was chosen over a quality-stratified sampling across all quality levels, nor do they discuss how this distributional bias might affect the conclusions drawn from benchmark comparisons.

    2.Subjective Annotations Without Agreement Reporting, and Unclear Clinical Utility: Pixel-level quality annotation is substantially more subjective than image-level labeling, particularly at the boundaries between Good/Usable regions, which are very ambiguous. Despite this, the paper reports no inter-annotator agreement metric (e.g., Cohen’s kappa or Fleiss’ kappa), which is a significant omission for a benchmark whose primary contribution is the quality of its annotations. More fundamentally, the practical utility of pixel-level quality maps for real-world CFP workflows remains unclear. While the localization of degradations is conceptually appealing, the paper does not provide any quantitative evidence that pixel-level quality information leads to measurable improvements in downstream clinical tasks (e.g., disease grading, screening accuracy) compared to image-level quality assessment. Without such evidence, the added burden of pixel-level annotation is difficult to justify from a clinical applicability standpoint.

    3.Limited Methodological Contribution and Training Conditions: EFIQA-CP’s contributions are relatively modest, consisting of two incremental modifications to the existing EFIQA: replacing a linear adapter with a wider CNN and substituting loss functions. Neither modification introduces conceptual novelty, and the ablation study reports only a single mDice value on the merged dataset, which is insufficient to fully characterize each component’s contribution. More critically, EFIQA-CP is trained on a private single-center dataset captured with a single device (Topcon Maestro 2), whereas the competing baselines are trained on different public datasets. This inconsistency in training data makes it impossible to attribute EFIQA-CP’s performance advantage to its architectural design rather than to a more favorable training distribution. A fair comparison would require either training all methods under identical conditions or explicitly analyzing the impact of training data choice.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission has provided an anonymized link to the source code, dataset, or any other dependencies.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    As discussed above, the paper is generally good as a small size testing CFP benchmark, however, it suffers from some major concerns. So I will wait for the rebuttal and give 3-weak reject in this stage.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors response to many concerns and state clear some of my questions. I have to say that I am statisfied but not enough (especially for size and Clinical utility).

    I will raise my score to accept but it is still weak accept.



Author Feedback

We thank reviewers for noting the relevance of FunPiQ as first pixel-level (PL) fundus image quality assessment (IQA) benchmark (R1,R2,R4); the comprehensive evaluation (R1,R2,R4); the failure-mode-driven design of EFIQA-CP (R1,R4); the use of multi-source, multi-device data (R4); and the principled visibility criteria (R4). We address reviewers’ concerns as prioritized by AC.

FunPiQ size (R2,R4) FunPiQ is a test-only benchmark, so all 300 images are used for evaluation. The size is consistent with or larger than established CFP datasets with expert PL labels: vessels (DRIVE:40, Leuven-Haifa:240), lesions (IDRiD:81, DDR:757), and optic disc (OD)/cup (Drishti-GS:101, RIM-ONE-r3:159). Annotation is challenging due to the expertise required to distinguish quality degradations from disease signs. Statistical robustness benefits from the paired comparison design, which controls for image-level variation.

Sampling strategy (R4) FunPiQ is designed to evaluate the localization of degradations by IQA methods (i.e. their explainability) task-agnostically (not mirroring specific prevalences), complementing image-level IQA benchmarks. Selecting image-level Bad images produces a rich mix of Good, Usable, and Bad regions (Fig.1B), so that each image tests the model’s ability to differentiate regions. This is common for PL tasks, where only images with objects of interest (in our case, degradations) are evaluated (e.g. IDRiD). We will add this rationale.

Clinical utility of quality maps (R2,R4) Quality maps offer advantages over global scores. Combined with anatomical localization, they enable dynamic, task-specific IQA without retraining—e.g. detecting periphery and OD visibility for DR screening and glaucoma. Maps also support deployment-time quality assurance, especially for non-experts, as they show where quality is insufficient rather than an opaque global score, aiding reacquisition.

Inter-annotator agreement (R4) We followed a consensus protocol: readers produced initial annotations on separate subsets under standardized criteria, which were then refined by a board-certified ophthalmologist. While this setup is common (e.g. IDRiD, DDR), it produces a single gold standard. We will report this as a limitation and planned extension.

EFIQA vs EFIQA-CP (R1,R2,R4) For ablation, EFIQA was trained on the same data as EFIQA-CP, so each component’s effect is isolated: EFIQA|+our CNN adapter|+nnPU (EFIQA-CP): mDice 62.67|63.36|65.10; QWK 63.26|67.00|68.41.Thus, the gain is due to architecture and loss, not data. The CNN adds the global context absent in EFIQA’s per-patch linear adapter, while remaining shallow to avoid overfitting. We chose CNN as CNN decoders (features→pixels) are standard for PL medical tasks (e.g. nnU-Net, SwinUNETR, TransUNet). ViT adapters are an interesting future direction. We will clarify these points, while adding an ablation table with all metrics.

SOTA comparison (R2,R4) Due to the different nature of the methods (end-to-end, explainable-by-design, anomaly detection [AD]), using the same training dataset would not be fair. Thus, we follow common benchmarking practices (e.g. KiTS and Learn2Reg MICCAI challenges) and use official weights or code when available or reimplement otherwise. AD methods, originally for industrial imaging, were tuned on high-quality CFPs (as required by their formulation) with broad disease diversity.

Implementation details and Fig.1 (R1) EFIQA-CP uses AdamW, LR=1e-3, 20 epochs, batch 16, input size 1024×1024, no augmentation, and nnPU (π=0.05). These details will be added. Code and weights are now at the anonymized repo. We substantially improved Fig.1, unifying the flow of data, stages, and examples.

Quantitative evaluation (R2) All metrics were computed by PL comparison against FunPiQ manual maps. This enables the quantitative paired evaluation in Tab.2.Fig.2 is complementary. We will make this explicit, and add mDice values to Fig.2 maps to show quantitative–qualitative alignment.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The paper introduces FunPiQ, a pixel-level fundus image quality assessment benchmark, and reviewers recognized its potential value for ophthalmic imaging and fine-grained quality localization. They also appreciated the inclusion of multiple baselines and the motivation to move beyond image-level quality assessment. However, several substantive concerns should be addressed in rebuttal. Reviewers questioned whether the 300-image benchmark is sufficiently representative, especially given the sampling strategy from images already labeled as poor quality. They also raised concerns about missing inter-annotator agreement, unclear clinical utility of pixel-level quality maps, limited evidence that the proposed method improves over EFIQA rather than benefiting from different/private training data, and incomplete implementation details. A focused rebuttal on these and other points raised by reviewers would help reassess the contribution.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper presents FunPiQ, a timely and useful benchmark for pixel-level fundus image quality assessment, together with EFIQA-CP as an explainable-by-design baseline. The reviewers recognized the value of moving beyond image-level FIQA toward localized quality assessment, and the rebuttal satisfactorily addressed the main concerns regarding dataset size, sampling rationale, quantitative evaluation, implementation details, and the comparison between EFIQA and EFIQA-CP. In particular, the authors clarified that the EFIQA/EFIQA-CP ablation used the same training data, reducing concerns that the improvement was mainly due to different training conditions, and provided missing implementation details to improve reproducibility. The paper’s clinical relevance is plausible, especially for localized quality control and image reacquisition, although reviewers noted that the clinical utility of pixel-level maps should be discussed more clearly. Overall, the rebuttal sufficiently mitigates the main concerns, and post-rebuttal reviewer opinions support acceptance. The final paper should clarify the benchmark’s intended use, annotation protocol/limitations, clinical utility discussion, and reproducibility details.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    Reviewers are broadly supportive of the paper and agree that the benchmark contribution is valuable. The rebuttal addressed the main concerns, especially around presentation and technical clarification, and the updated reviewer opinions remain positive. One reviewer did not provide a post-rebuttal update, but the available evidence still supports a positive overall assessment.



back to top