List of Papers Browse by Subject Areas Author List
Abstract
Machine learning models applied to resting-state functional MRI (rs-fMRI) data have reported increasingly high classification accuracy for Autism Spectrum Disorder (ASD) diagnosis, with some recent works claiming accuracies above 97%. In this paper we investigate whether such figures reflect genuine generalisation or are artefacts of data leakage introduced during feature selection. We conduct a systematic reproducibility study of two peer-reviewed ASD classification pipelines built on the public ABIDE~I dataset: \emph{MADE-for-ASD} and {XAI-for-ASD}. We identify and correct concrete leakage faults in both codebases — global F-score-based feature selection prior to cross-validation in the former, and pre-fold application of supervised Recursive Feature Elimination (SVM-RFE) in the latter — and quantify the resulting performance inflation. After correction, accuracy drops from 74.04% to 66.35% for MADE-for-ASD and from 97.68% to 65.02% (±2.51%) for XAI-for-ASD, revealing gaps of up to 33 percentage points. Independent corroboration from Dong et al., who corrected a separate leakage fault in EV-GCN on ABIDE~I and observed a drop from approximately 81% to approximately 68%, provides convergent evidence that leakage-induced inflation is a recurring rather than isolated occurrence in this literature, though its broader prevalence across the field remains to be established. Beyond diagnosis of the problem, we provide corrected, publicly available forks of both codebases and propose practical safeguards. We further argue that large language model (LLM)-based code review tools, used here as a first-pass diagnostic aid complementing expert validation, represent a cost-effective quality-assurance layer for research software.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/5823_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/mhdbashar/XAI-for-ASD
https://github.com/mhdbashar/MADE-for-ASD
Link to the Dataset(s)
N/A
BibTex
@InProceedings{DisMoh_Inflated_MICCAI2026,
author = { Disoki, Mohamad Bashar AND Sonbol, Riad},
title = { { Inflated Performance in fMRI-Based ASD Diagnosis: A Reproducibility Study of Data Leakage in Feature Selection Pipelines } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16886},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
Through two concrete case studies, the authors demonstrate that data leakage in the feature selection pipeline can substantially inflate reported performance in fMRI-based autism spectrum disorder (ASD) classification. By reproducing and correcting methodological flaws in two previously published studies, they show that the originally reported accuracies of 97.68% and 74.04% drop markedly after proper correction, bringing the results closer to established benchmarks in the field. This work serves as an important methodological caution for researchers in neuroimaging-based machine learning and is further strengthened by the release of corrected code and practical recommendations for avoiding similar errors.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper highlights a critical methodological issue that directly affects the credibility of findings in neuroimaging-based machine learning research.
- The study follows a convincing and transparent workflow of reproduction, correction, and comparative re-evaluation, with clearly quantified results. This enhances the paper’ s persuasiveness.
- The authors make their corrected code publicly available, which improves the reproducibility and practical value of the work.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- The level of methodological novelty is somewhat limited. The study mainly verifies and applies already recognized methodological principles, rather than introducing a fundamentally new concept or technique.
- The evidence supporting the generality of the conclusions remains insufficient. Although the warning raised by the authors is reasonable and important, drawing broad implications for the field from only two case studies leaves the empirical basis somewhat limited. Stronger support for claims of broader prevalence would require a wider survey or more systematic analysis.
- Some experimental details and analyses need to be described more thoroughly. In particular, the setup of the corrected experiments, especially the treatment of hyperparameters, is not sufficiently detailed, which weakens the rigor of attributing the observed performance drop solely to data leakage.
- The current work focuses primarily on correcting and comparing two published cases. The paper would be substantially strengthened by either screening a larger number of potentially affected studies or providing a deeper analysis of the interaction between data leakage and hyperparameter choices.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This study addresses a real and important methodological issue and provides two illustrative case studies with clear cautionary value for the field. However, in its current form, the manuscript does not yet fully meet the standard for publication in terms of the level of innovativeness, the strength of support for its broader claims, and the depth of experimental analysis.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have provided a reasonable and constructive response to the main concerns. Although the methodological novelty remains limited, the paper makes a valuable reproducibility contribution by identifying concrete data-leakage faults in two published ASD-fMRI classification pipelines and quantifying their impact after correction. The authors’ decision to retain the original hyperparameters in the corrected experiments is methodologically defensible, as re-tuning would introduce an additional source of variation and obscure the isolated effect of leakage. They also appropriately acknowledge that two case studies alone cannot establish field-wide prevalence and commit to moderating the broader claims.
Some concerns remain, particularly the need to phrase the attribution of the XAI-for-ASD performance drop more carefully, since several corrections were applied simultaneously. The claims regarding AI-assisted code review should also be presented more cautiously. However, these issues can be addressed through clearer reporting and more careful framing. They do not invalidate the main empirical findings. Given the importance of the identified leakage problems and the practical value of the corrected pipelines and safeguards, I would recommend acceptance.
Review #2
- Please describe the contribution of the paper
1.This work systematically identifies specific data leakage issues in two existing ASD classification pipelines, clearly tracing and analyzing their sources, and proposes corresponding correction strategies. 2.Based on the corrected pipelines, the paper provides improved implementations, offering a more reliable experimental baseline for subsequent studies. 3.The authors summarize and propose a practical checklist for detecting data leakage in machine learning-based fMRI classification tasks, which is actionable and of clear practical value. 4.The paper further explores the potential of incorporating AI-based static code analysis into the peer review process, positioning it as a complement to conventional reviewing to enhance experimental reliability and code quality control.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses a critical yet often overlooked issue in neuroimaging-based machine learning research, namely data leakage, which is essential for improving the reliability and reproducibility of results. 2.Beyond identifying the problem, the authors distill a generalizable checklist that provides practical guidance for similar fMRI classification studies. 3.By re-evaluating existing pipelines, the study reveals potential sources of performance inflation, offering valuable insights for the community. 4.The idea of integrating AI tools into the code review process aligns well with current trends toward reproducibility and automated auditing, and shows a degree of forward-looking thinking.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.After correcting the data leakage issues, model performance drops noticeably. However, it is unclear whether a systematic hyperparameter search and re-optimization were conducted on the revised pipelines, making it difficult to assess whether the reported results reflect their best achievable performance. 2.The paper lacks a comparative analysis of discriminative features (e.g., key brain regions) before and after correction. In particular, it remains unclear whether the neurobiological interpretations reported in prior work are preserved, which is important for validating the method. 3.The discussion on AI-assisted code review remains largely conceptual. It would benefit from more concrete details, such as general prompt templates or semi-automated review workflows, to improve practicality and reproducibility. 4.The experiments are primarily conducted on a single dataset (e.g., ABIDE I). Additional cross-dataset validation (e.g., ABIDE II) would strengthen the robustness and generalizability of the findings. 5.While the proposed checklist appears general, empirical validation is limited to ASD tasks. Including case studies on other psychiatric disorders or similar classification problems would further support its broader applicability. 6.There are several formatting issues in the submission, such as tables and text exceeding page boundaries (e.g., table sizes and content on page 3), as well as inconsistent header formatting. These should be corrected in the final version.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper investigates data leakage in machine learning-based fMRI classification pipelines, systematically identifying and correcting critical flaws in two representative ASD pipelines. The results clearly demonstrate the substantial impact of data leakage on performance evaluation. Building on this, the authors propose a general-purpose checklist for detecting leakage and explore the use of AI tools for code auditing, both of which are valuable for improving research rigor, reproducibility, and result credibility in the neuroimaging community. From a methodological perspective, the study is reasonably sound and shows potential for broader applicability across related tasks. However, the current version would benefit from a more complete characterization of the corrected pipelines (e.g., hyperparameter settings and feature interpretation), a more concrete realization of AI-assisted review tools, and expanded experimental validation. In addition, formatting issues should be addressed. Overall, this work represents a practically relevant methodological reflection with clear guidance value. With the above issues addressed, it has the potential to make a meaningful contribution to the community.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The paper examines two published rs-fMRI-based machine learning approaches for ASD classification to assess whether their reported accuracies reflect genuine generalization. The authors identify data leakage issues in both approaches and present corrected implementations, which yield substantially lower performance. The paper also provides recommendations for avoiding such issues.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Relevance: Data leakage and overestimation of performance are well-recognized issues in the machine learning literature. Addressing these problems in the context of rs-fMRI-based ASD classification and providing concrete recommendations makes the contribution valuable.
- Transparency and reproducibility: The authors provide corrected implementations of the evaluated methods, enabling a clearer assessment of their true performance and supporting reproducibility.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Limited novelty: While the paper addresses an important issue, the methodological novelty is limited. The proposed contributions consist primarily of relatively minor corrections to previously published pipelines, without introducing new analytical frameworks or substantially extending prior approaches.
- Inconsistent and incomplete reporting of results: The presentation of results lacks consistency. For example, Table 1 reports performance as decimal values, whereas Table 2 uses percentages. In addition, although both analyses employ cross-validation, standard deviations are only reported for the corrected results in the second case study. This inconsistency makes it difficult to interpret variability and compare results across experiments.
- Limited depth of discussion: The discussion does not fully explore the broader implications of the findings. For instance, the claim that leaderboard results are often above 90% and likely overestimated is important, but its impact on the field is not sufficiently analyzed. Similarly, while the authors suggest that data leakage may affect previously reported biomarkers, they do not provide additional analyses to support or quantify this claim.
- Weak integration of the clinical context: The introduction of the medical domain is somewhat superficial and occasionally imprecise. For example, statements such as describing ASD as a “prevalent condition” or implying that rs-fMRI directly measures “whole-brain functional connectivity” would benefit from more careful wording and stronger grounding in the neuroimaging literature.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- Justification of baseline methods: The rationale for selecting the baseline methods used for comparison is unclear. Given the wide range of reported performances in the literature, it would be important to explain why these specific methods (with accuracies around ~70%) were chosen and how representative they are of the state of the art.
- References: The use of references in the abstract is unconventional. In addition, the ABIDE dataset paper is cited when introducing ASD and rs-fMRI, but not consistently when the dataset itself is introduced later.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
While the topic is important and the intentions are aligned with improving research practices, the current version does not yet meet the bar for acceptance. That said, the paper is close to the threshold, and with clearer experimental design, stronger discussion, and improved presentation, it could become a solid contribution.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The rebuttal adequately addressed the main concerns raised in the initial review. The authors acknowledged the identified limitations, clarified points of ambiguity, and committed to concrete revisions, including extending relevant discussions, moderating claims, and adding supporting references. These are reasonable improvements that, taken together, bring the paper to an acceptable level.
Although the authors justified the formatting differences across tables as a deliberate choice to match the original replicated papers, I expect that at least the formatting issues will be adjusted in the final version, and I would encourage the authors to add the standard deviations in Table 1. Still, the rebuttal demonstrated a clear understanding of the paper’s weaknesses and a credible plan to address them. Contingent on the promised revisions being carried out, I support acceptance.
Author Feedback
We thank the reviewers and meta-reviewer for their constructive feedback. We address each major concern below and commit to concrete revisions.
[R1, R2, Meta] Hyperparameter treatment We intentionally retained the original hyperparameters in the corrected experiments. Re-tuning after removing leakage would introduce a second source of variation, making it difficult to attribute performance changes specifically to leakage. Our goal is to isolate the effect of leakage, which requires holding all other factors constant. We will clarify this rationale explicitly in the revision.
Importantly, the corrected accuracies (66.35% and 65.02%) are not anomalously low. Dong et al. (Human Brain Mapping, 2025), an independent study on ABIDE I using a standardized framework with per-fold tuning across multiple architectures, reports test accuracies in the roughly 65–72% range. Our results fall within this interval, supporting the interpretation that they reflect realistic generalisation rather than under-optimised models.
[R1, R3, Meta] Scope of case studies and strength of claims We agree that two case studies alone are insufficient to establish field-wide prevalence. Accordingly, we will moderate our claims in the abstract and conclusion, replacing broad generalisations with more precise, evidence-backed statements.
At the same time, independent corroboration strengthens this point. Dong et al. (2025) explicitly state in Section 2.5.2 that they adapted EV-GCN from Huang and Chung (2020), resolved an information leakage issue in the EV-GCN model, and performed fold-wise model selection using validation performance. Under this corrected setup, EV-GCN drops to about 68%, far below the originally reported 81%. Together with our two case studies, this provides converging evidence that leakage-induced inflation recurs across distinct ABIDE I pipelines. We will incorporate this reference and adjust our framing to reflect this broader but still carefully scoped evidence.
[R2, Meta] AI-assisted code review The workflow is described in Section 6.2: we use structured prompts derived from the checklist in Section 6.1, such as identifying supervised feature selection applied outside cross-validation loops, forming a reusable QA protocol. We emphasise that AI review is intended as a first-pass diagnostic tool, not a substitute for expert validation. To improve clarity, we will add a table of example prompts and explicitly scope the role of AI in the revision.
[R2, Meta] Neurobiological interpretability This concern is already addressed in Section 7 (“Broader implications for reported biomarkers”): the XAI-for-ASD interpretability analysis identifying visual-cortex markers was performed on a model whose feature space was pre-optimized using test-set labels, meaning those biomarker attributions lose statistical validity after correction. While recomputing feature importance under leakage-free pipelines exceeds the scope of this reproducibility study, we note that Dong et al. (2025) report leakage-free reanalysis and feature-importance results in broadly comparable regions. We will expand this into a dedicated discussion paragraph, clearly scoping interpretability as future work.
[R3] Clinical framing and terminology We will revise the introduction to use precise neuroimaging terminology, clarify that rs-fMRI measures BOLD-derived functional connectivity, replace imprecise prevalence statements with the cited WHO figure, and ensure consistent ABIDE citation throughout.
[R2, R3, Meta] Table reporting consistency The differing formats in Tables 1 and 2 are intentional: Table 1 uses point estimates to match MADE-for-ASD’s original single-run reporting, while Table 2 reports mean ± std to match XAI-for-ASD. This design enables direct, like-for-like comparison with each paper’s reported results. We agree this was not sufficiently explained and will add clarifying notes in the table captions.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Data leakage is an important issue that can be overlooked and is often ignored. It leads to overestimation of the results and questionable exagerations of clinical impact. This is the starting point and motivation for this proposal and, this motivation itself, is one of the strengths of the paper that all reviewers agree with. Not just that, but the authors provide corrected implementations and a checklist of good practices. This is also underscored by all the reviewers.
However, there are also clear limitations in the current version of the manuscript. In what follows are a list of potentially adressable weaknesses informing my decision.
- The number of case studies is limited (2) which does not guarantee the broad implications presented by the authors [R#1]. This should be backed up by a larger set of examples and a more systematic approach (probably impractical without new experiments) or toned down conclusions.
- Related to that, there should be a more in-depth discussion [R#3] with more details on hyperparameters [R#1 and 2] and AI-use [R#2] and ideally some details on how the methodology fixes impact neurobiological interpretations [R#2].
- The format should be carefully checked. There are clear inconsistencies between the results tables that should not be present in a final draft [R#2 and 3]. This not only difficults comparison but also detracts from the final presentation, making the whole manuscript feel unpolished.
Nonetheless, this is a non-exhaustive list of things to address and I would advise the authors to check all the specific requests from each reviewer.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
I think the authors did a really good job of being transparent and open to downplay some of the claims which were the things I was looking forward with the rebuttal. This is not just my opinion but also the opinion of reviewers #1 and #3 (there was no justification given by reviewer #2).
In that sense, as long as the authors include all the information and promises from the rebuttal in the final manuscript, I see no issues with the submission.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Based on the clarifications provided in the rebuttal, a majority of the reviewer’s concerns have been addressed. I believe this is a timely and interesting contribution which addresses an important gap in reporting pipelines and studies contributing factors systematically and adds value to the broader neuroimaging community
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper presents a reproducibility evaluation of ASD classifiers under data leakage and explores changes in performance after correcting this leakage using AI-based code review in two studies. The reviewers pointed out the limited novelty of the results, a point with which I partially disagree, especially if we consider this an application study. Secondly, the reviewers raised several weaknesses, including the absence of hyperparameter tuning in the reproduction after leakage correction, and requested more details on the AI-based code analysis used to detect leakages. The authors’ rebuttal addresses these two points perfectly. The reviewers seem satisfied with the authors’ responses to their concerns and acknowledge that the authors are particularly aware of the paper’s limitations and are willing to make appropriate changes to address them. Given the quality and rigor of this study, especially given the complexity of reproducing published findings, I recommend acceptance.