List of Papers Browse by Subject Areas Author List
Abstract
Reproducibility policies promise “checkable” medical-imaging science, yet many submissions still ship unverifiable artifacts. Our analysis of 3722 MICCAI papers shows code-linking rising from 51.8% (2021) to 72.5% (2025), but ~13% of linked repositories are inaccessible or empty. We present paper-snitch, a reviewer-facing decision-support tool that turns these signals into an evidence-grounded report. Paper-snitch parses PDFs, resolves and sanity-checks repositories, and applies policy-aware checklists aligned with MICCAI expectations, producing a review-time verifiability score decomposed into interpretable sub-scores plus criterion-linked excerpts and artifacts reviewers can inspect. It never executes untrusted code or attempts GPU-heavy reproduction, focusing instead on bounded, verifiable checks. We compare paper-snitch on 100 randomly sampled MICCAI 2025 papers with human annotators using shared evaluation criteria, indicating that automated, bounded checks can scale reproducibility screening while keeping final decisions with reviewers.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/0729_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/AImageLab-zip/paper-snitch
Link to the Dataset(s)
N/A
BibTex
@InProceedings{BolFed_The_MICCAI2026,
author = { Bolelli, Federico AND Santoli, Davide AND Marchesini, Kevin AND Lumetti, Luca AND Grana, Costantino},
title = { { The Paper Has a GitHub, the GitHub Has a README, the README Has Nothing: Reproducibility Signals for Review Support } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper proposes a tool to analyze papers in support of peer-review. While it doesn’t fall within the traditional scope of MICCAI contributions, it addresses the important problem of reliability and thoroughness of reproducibility reviews during the peer-review process.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Open a discussion on the sustainability and quality of peer review
- Propose a LLM-based tool to analyze papers and associated code repositories, with acceptable performance compared to human reference
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Casual writing tone
- Lack of comparison with other LLM-based reviewing tools
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important topic: the reliability of reproducibility evaluations during peer review. It describes and evaluates a LLM-based tool to produce reproducibility checklists, dataset and code analyses.
The paper doesn’t compare the proposed system to any baseline, nor does it reference related tools on LLM-based reviews. For instance, it would be important to understand the specific relevance of the tool presented compared to existing AI agents/companions such as Claude Code, Gemini CLI or Open AI Codex, and a reference to previous literature on LLM-assisted peer review (e.g., https://aclanthology.org/2024.emnlp-main.70).
Relatedly, LLM-assisted peer review raises a number of questions that are not discussed by the authors, including:
- Privacy of the authors’ submissions;
- Biases in human and LLM evaluations;
- Prompt injections in submissions. Given that the main contribution of the paper isn’t directly in the MIC or CAI domains, it would be important to discuss the broader implications of using such technical solutions in the review process.
The tone of the writing is slightly casual, including phrases such as “Let us start by lowering our blood pressure”, which I think should be avoided. Similarly, the name of the tool, paper “snitch”, suggests a confrontational approach to peer review, which may raise questions.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The authors present Paper-Snitch, a decision-support framework designed for reproducibility screening during the peer-review process for conferences. Unlike automated filters, the system focuses on artifact verifiability through three core pillars: (i) evidence-grounded outputs traceable to specific PDF spans or repository files, (ii) review-time safety via static inspection (avoiding untrusted code execution or heavy GPU training), and (iii) a scalable, auditable workflow for conference-level deployments. The system elegantly separates probabilistic LLM-based fact-detection from a deterministic mathematical scoring module, providing both a numerical grade and actionable recommendations for authors.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Targeted Utility: The paper addresses a critical, growing need in the MICCAI and broader academic community for standardized, scalable reproducibility checks during conference reviews.
- Deterministic Actionable Outputs: The framework’s decision to use LLMs only for finding evidence while leaving the actual scoring to fixed mathematical formulas is an elegant solution to the problem of LLM non-determinism. Moreover, by generating specific recommendations (e.g., flagging missing checkpoints or dataset links), the tool moves beyond a simple “pass/fail” gate and can serve as a constructive feedback mechanism for authors.
- Static Safety: The reliance on static inspection rather than code execution ensures the tool is secure and computationally efficient enough to be feasible for large-scale conference use.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- LLM Specification Ambiguity: There is a lack of clarity regarding which models were used. While Section 4 and the code repository mention GPT-5, Table 1 suggests GPT-4o performed better. The paper would benefit from clarifying which model was ultimately used and referencing the choice in LLM earlier than Section 4.* Fragility of Code Analysis: The “Code Embeddings” section appears highly dependent on specific file-naming conventions (e.g., “train/test”). The authors acknowledge this fragility, but the current implementation may fail on repositories with non-standard structures or unclear naming, which may be equally likely at conferences. Furthermore, how does Paper-Snitch handle Jupyter notebooks?
- Free/Open-Source Alternatives: The paper does not address why proprietary models (OpenAI) were chosen over open-source alternatives (e.g., Ollama).
- Nitpick: There appears to be an overuse of italics for serving different functions throughout the paper.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- It would be useful to clarify the LLM notation in the paper and what it refers to and the other comments brought up in the “Weaknesses” section
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The recommendation is primarily driven by the framework’s ability to successfully address an urgent gap in standardized reviewing criteria for conference-level reproducibility. By focusing on artifact verifiability, the authors provide a scalable, “review-time safe” solution that is critically needed in the MICCAI community. A key contribution lies in its deterministic scoring logic, which resolves LLM non-determinism by restricting the model to evidence-gathering via a structured checklist. Moreover, the authors not only address a reproducibility concern, but make their code and a working demo readily available.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
This paper introduces a tool called Paper-Snitch for assessing reproducibility and verifiability in medical imaging papers, especially MICCAI. Instead of trying to fully reproduce results, the tool performs bounded checks that are feasible during peer review: it parses the paper, identifies the paper type, inspects linked repositories without executing code, evaluates paper/code/dataset artifacts against policy-aligned criteria, and produces a verifiability report with scores, evidence snippets, and qualitative summaries. The authors motivate the work with an analysis of 3722 MICCAI papers showing that although code sharing has increased, a large fraction of linked repositories remain empty. They evaluate the system on 100 MICCAI-2025 papers against experienced human annotators and report high percentage of machine-human agreement.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.This paper addresses an important problem and has a clear scope. It is important to check reproducibility during peer review, but artificially checking the code is difficult and time-costing for reviewers.
2.The overall system design is thoughtful and well structured since it separates paper analysis, code analysis, dataset analysis, and final aggregation into interpretable components, and it grounds its outputs in inspectable artifacts.
3.The paper is well written and generally easy to follow. The authors do a good job explaining what the system is supposed to do. The figures and diagrams are clearly presented.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Several parts of the pipeline rely heavily on LLM-based decisions, including classification, criterion assessment, and repository file selection. Although the paper is transparent about it in the limitations section, it is still a concern whether the proposed system can be implemented in a trustworthy way that ensures privacy, which is important in peer-review process.
2.The evaluation is somewhat limited since it is conducted on 100 papers from a single venue (MICCAI-2025). This leaves some uncertainty about robustness and generalizability.
3.It would strengthen the paper if the authors included a self-case-study showing how Paper-Snitch evaluates this submission itself and its accompanying repository, since this would help readers understand the form and usefulness of the generated report in a concrete end-to-end example. (This is not a weakness, but something interesting to see.)
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper studies an important problem, has a reasonable idea and implementation, and is well written. My only concern is practicality (generalizability, privacy, etc.) of using it in peer-review.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank the reviewers and meta-reviewer for their constructive feedback. We are encouraged that they identified several strengths: the paper addresses a timely problem for MICCAI, opens discussion on the sustainability and quality of peer review, and proposes a practical reproducibility-screening framework. Reviewers appreciated that Paper-Snitch is not an automated reviewer, but an auxiliary tool for verifiability. They also highlighted evidence-grounded design, static inspection without executing untrusted code, interpretable analyses, deterministic scoring, actionable recommendations, human agreement, and available code/demo resources.
Scope and role in peer review. Paper-Snitch is intended only to support reviewers or chairs with auditable checks of objective artifact signals; final judgment remains human. We will strengthen this framing.
Privacy. Submissions cannot be sent to external AI services without policy, consent, and processing guarantees. The current implementation is a proof-of-concept; the framework is model-agnostic. There are multiple options to evaluate when moving to production: running the pipeline in a controlled environment, disabling external web search during double-blind review, using local/open-weight models, storing minimal audit artifacts, or ensuring that manuscripts and repositories are not used for model training with a specific contract signed with service providers.
Regarding bias, prompt injection, and robustness, LLM-assisted peer review can introduce risks. Paper-Snitch mitigates some of them by restricting LLMs to evidence retrieval and structured criterion assessment, while scores are computed deterministically. This does not eliminate bias, but provides tools for mitigation. Like medical imaging tools that assist expert clinicians and surgeons in daily clinical workflows—supporting, standardizing, and accelerating specific tasks without replacing professional judgment—Paper-Snitch is designed to play an analogous role in the peer-review process. It provides reviewers and chairs with structured, evidence-grounded reproducibility checks, helping them identify missing or unverifiable artifacts more consistently while leaving all scientific and acceptance decisions to human experts. For prompt injection, schema-constrained outputs and deterministic tree checks are used. We avoid executing untrusted code.
Evaluation. 100 MICCAI 2025 papers are an initial validation rather than definitive evidence of generalizability. We will clarify that the papers were randomly sampled and stratified by method, dataset, and both.
Model choice and baselines. Both GPT-4o and GPT-5 were evaluated under the same embeddings and scoring code. Notably, the demo is designed with extensibility in mind, and models powering the tool can be configured at runtime. Future extensions will include a thorough evaluation of open-source and local alternatives, including Ollama-served models, alongside agentic tools such as Claude Code, Gemini CLI, and OpenAI Codex.
Regarding code-analysis fragility, non-standard repositories may challenge file selection. We will clarify this limitation and note that repository templates or required entry points would improve reliability. Jupyter notebooks are handled statically as JSON/text artifacts: markdown, imports, cells, commands, and outputs can be inspected, but notebooks are not executed.
Finally, we clarify the provenance of Fig. 1. The MICCAI code-availability statistics are part of this study and are produced by the code-availability node applied to the analyzed MICCAI corpus.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper addresses a timely and important topic: assessing the reproducibility of papers during conference review periods. The authors propose an interpretable framework, largely based on LLMs and paid services, to analyze various aspects of papers and compute mathematical scores aligned with standard reproducibility checklists in conference review forms. The reviewers agreed on the paper’s main strengths: it opens a discussion on the sustainability and quality of peer review, and the framework provides acceptable performance compared to human reference. However, R#1 and R#3 pointed out important points that should be addressed in the rebuttal:
- The proposed framework is based on online tools, which can raise concerns about privacy issues. During conference reviews, confidentiality prevents reviewers from feeding the full PDF paper to any AI Agent. How does the proposed framework tackle this issue?
- Even if the authors demonstrated a strong agreement between human & LLM evaluations, it is important to discuss the potential biases that could arise from the use of LLM: i) How could we incorporate such a framework for peer-review without introducing biases in the evaluation process? ii) How could we ensure its robustness towards prompt injection or AI-generated papers?
- The evaluation of the proposed framework is limited to 100 papers, randomly sampled from MICCAI 2025 across 3 categories. Could you clarify the distribution of reproducibility scores across these papers? Did you use any form of stratification for sampling these papers?
- In addition, only two architectures for the LLM have been compared (GPT4-o and GPT5). Comparison with existing AI agents such as Claude Code, Gemini CLI or Open AI Codex, and a reference to previous literature on LLM-assisted peer review would be needed.
As a minor point, the authors should clarify the provenance of the results shown in Fig. 1.Do these metrics come from a previous study, or is it part of this study? If so, the authors should give more precisions about it.
