List of Papers Browse by Subject Areas Author List
Abstract
As technologies for applications in medical imaging and computer-assisted interventions mature, technical feasibility alone is insufficient for translation to bedside. Highlighted by communities that emphasize human-centered technology design and recent efforts at MICCAI, there is a need to evaluate technological solutions with human involvement. Among the various forms of human-centered evaluation, we focus on empirical, controlled user studies of human-technology interaction. These evaluations involve actual applications and real end-users, providing actionable evidence about human impact and intended use. However, the lack of standardized studies and unique challenge in user studies complicate the measurement of progress and reproducibility in research. This paper aims to support the MICCAI community in conducting evaluations with human subjects, by drawing on insights from human-centered principles and quantitative research in human-computer interaction. We discuss key considerations that include the formulation, planning, and execution of valid and meaningful experiments. The rigor and use of systematic methods in human-subjects evaluations will ultimately provide researchers with more reliable evidence to assert the potential achievements of their technological solutions. As considerations for human-subject experiments differ from those for technical development, the evaluation of these works demands criteria within the MICCAI community that account for the substantial effort and rigor required to conduct human-subject evaluations. Findings and lessons from human-machine interaction studies will guide future research aimed at deploying technologies and assessing clinical effectiveness.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/6182_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{ChoSue_Research_MICCAI2026,
author = { Cho, Sue Min AND Gomez, Catalina AND Breininger, Katharina AND Creighton, Francis AND Guo, Xiaoqing AND Ho, Dean AND Ishii, Masaru AND Jannin, Pierre AND Kersten-Oertel, Marta AND Kim, Seong Tae AND Navab, Nassir AND Ouyang, Cheng AND Wu, Shandong AND Yi, Paul AND Zuluaga, Maria A. AND Unberath, Mathias},
title = { { Research Design Considerations for Empirical User Studies in MICCAI } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16895},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper presents a methodological framework for conducting empirical, controlled user studies in medical imaging. It adapts the “Four Validities” (statistical, construct, internal, and external) from the HCI field to the specific constraints of MICCAI, providing a structured guide for researchers to plan and execute human-subject experiments.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper offers an exceptionally clear and detailed translation of the four validities (internal, external, construct, and statistical conclusion) into the medical context, which is often neglected in algorithm-focused papers. 2.It provides a comprehensive playbook for researchers, covering critical design elements such as variable control, participant selection, and bias mitigation. This is highly valuable for bridging the gap between benchtop engineering and bedside validation. 3.As the community moves toward human-centered AI and AR systems, this paper sets a standard for how these systems should be evaluated, potentially reducing the prevalence of poorly designed pilot studies.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.While the theoretical framework is sound, the paper lacks a concrete worked example or a case study demonstrating how these guidelines can be applied to a specific MICCAI problem (e. g. , evaluating an AI-based diagnostic tool). 2.Additionally, the significant conceptual overlap with another submission is problematic. It appears the authors have split a single contribution into two manuscripts.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The manuscript provides a ‘playbook’ for user studies, which is a weak spot in the MICCAI community. The adaptation of the four validities (statistical, construct, internal, external) to the surgical/imaging context is well-executed. However, my score is capped at 4 because the guidelines remain theoretical. To be truly ‘Good’, the authors should have demonstrated these principles on a concrete benchmark. It’s a useful tutorial-style paper, but not a breakthrough.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
The paper provides a structured set of research design considerations for conducting empirical user studies in medical image computing and computer-assisted interventions. Its main contribution is to formalize key aspects such as study goals, system design, user tasks, participant recruitment, and evaluation metrics, while highlighting the importance of human-centered validation alongside technical performance. It aims to guide the MICCAI community toward more rigorous, reproducible, and clinically relevant human-subject evaluations.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The primary strength of the paper lies in its timely and important focus on human-centered evaluation, which is often underrepresented in MICCAI research. It clearly identifies gaps in current practices and provides practical, well-structured guidance for designing controlled user studies. The discussion is comprehensive, covering multiple stages of study design from planning to execution, and effectively bridges concepts from human-computer interaction and medical imaging. Another strength is the emphasis on reproducibility, transparency, and real-world clinical relevance, which enhances the paper’s impact and applicability. The inclusion of practical considerations and real challenges, such as recruitment and measurement validity, makes the work valuable for researchers planning empirical studies.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The paper is largely conceptual and does not introduce a novel technical method or empirical validation, which may limit its perceived contribution compared to more technical submissions. While it provides useful guidelines, many of the discussed principles are adapted from established literature in human-computer interaction and psychology, and the paper could better highlight what is uniquely tailored to MICCAI. Additionally, the lack of concrete case studies or experimental results makes it difficult to assess the practical effectiveness of the proposed recommendations. Some sections are also dense and could be simplified for improved readability.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper addresses an important and often overlooked aspect of medical AI research by emphasizing the need for rigorous human-centered evaluation. Its contribution lies in consolidating best practices and encouraging the community to adopt more structured and transparent study designs. Although it lacks technical novelty and empirical validation, its potential impact on improving research quality and clinical translation justifies a positive recommendation.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The primary contribution of this work is a set of considerations designed to improve the quality and rigor of controlled user studies. It advocates for the integration of human-centered principles and Human-Computer Interaction (HCI) methodologies into research workflows. These considerations are systematically organized into five key aspects of study design: Study Goals, Technology Design, User Tasks and Interfaces, User Recruitment, and Measurement and Evaluation. The work is specifically tailored for the MICCAI community, addressing the unique challenges and requirements of medical image computing and computer-assisted interventions. It serves as a qualitative resource to help researchers bridge the gap between technical development and human-centric validation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-
Clinical Relevance: The paper addresses a highly relevant topic, since human-centered development is essential for successfully translating research into clinical practice. This mirrors the cited recent evolution in Explainable AI (XAI), where the integration of HCI principles is becoming indispensable for making model explanations truly actionable in medical settings.
-
Critical Methodological Reflection: The work provides a critique of current practices, such as the reliance on crowdsourcing platforms like Prolific for user recruitment. It correctly highlights the potential gap between high-quality data collection in controlled environments and the actual ecological validity required for real-world clinical applications.
-
Transparency and Reproducibility: The paper promotes a culture where limitations are clearly recognized and research can be more reliably reproduced. The call for greater transparency in reporting human-based evaluations is a significant strength. The community is currently open to sharing code and should likewise commit to sharing the specific details of the conditions for each human evaluation. This will allow us to recognize its limitations.
-
Structured Guidance: The organization of considerations into five clear dimensions (goal, technology design, user task, recruitment, and measurement) provides researchers with an intuitive and systematic way to audit their own study designs.
-
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Abstraction: While the proposed considerations are conceptually sound, they occasionally remain at a high level of abstraction. The practical utility of the work would be significantly enhanced if the authors provided a more concrete “action plan” or a case study demonstrating how to operationalize these qualitative guidelines within a specific MICCAI-related technical project.
-Trade-offs: The paper advocates for HCI practices without a sufficient discussion on the subsequent costs and trade-offs. For instance, while pilot studies are essential for identifying protocol flaws and technical bugs, they extend research timelines and resource requirements. A discussion on how to balance these gold-standard principles with the practical constraints of typical research cycles would be beneficial. Furthermore, while it is true that platforms like Prolific often collect data under conditions that may differ from those expected in real-world applications, it is equally important to address the significant practical difficulties involved in recruiting a user pool specifically qualified for medical tasks, as well as the logistical complexity of organizing experimental scenarios that closely mirror actual clinical environments.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
While Figure 1 is highly illustrative and helpful in summarizing the proposed guidelines, the inclusion of subsection numbers (i.e., 2.1, 2.2, …) within the graphic is confusing in a visual context. I suggest removing these numerical references to provide a cleaner presentation.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I recommend a Weak Accept. The paper is relevant because it discusses how to properly involve humans in medical AI research. I believe this work is useful because it encourages the community to conduct better studies and be more transparent about how they test their tools in real scenarios. Furthermore, this work creates an opportunity for discussion about the validity and limitations of human-based evaluations. However, the main problem is that the advice is overall too general. It is difficult to see how a researcher could use these rules in a real-world project without more specific examples. Also, the paper does not discuss the practical costs, such as the extra time or money needed for some of the discussed steps. Despite these issues, the discussion it starts is very valuable, which is why I suggest accepting it.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers for their constructive feedback. We are encouraged by the recognition of our work’s importance in MICCAI, including its relevance to human-centered evaluations and their impact (R1, R2), the five key aspects highlighted (R1, R2, R4), and its clinical relevance (R1, R4).
Below, we grouped main concerns identified both in the meta-review and in the reviewer’s comments, and we responded to how these will be addressed in the camera-ready version.
<Abstract and conceptual framework (R1, R2, R4)> We agree that demonstrating the framework’s practical utility would strengthen its value and better ground the proposed considerations for conducting empirical user studies. Therefore, we will present a case study based on a MICCAI publication from previous years. Through this example, we will illustrate how the considerations apply to methodologies previously used to design and evaluate medical systems with human involvement. For example, consider publication “You, F., Khakhar, R., Picht, T., & Dobbelstein, D. (2020). VR simulation of novel hands-free interaction concepts for surgical robotic visualization systems. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 440-450)”.
<Novelty and contribution (R2)> We believe the value of our paper lies in its potential impact on improving research quality and clinical translation in MIC and CAI. The principles are intentionally adapted from established HCI and psychology literature, and this grounding is precisely what makes them reliable and actionable. However, these principles require tailoring to MICCAI’s unique constraints: specialized clinical populations, safety-critical applications, physical system interactions (surgical navigation, robotics, phantoms), and domain-specific measurement challenges. We will clarify these MICCAI-specific adaptations more explicitly in the manuscript.
<Discussion of practical considerations (R4)> We will add discussion acknowledging that pilot studies extend timelines, clinician recruitment involves significant logistical challenges, and high-fidelity simulations require substantial investment. We will address how researchers can prioritize considerations given practical constraints.
< Overlap with another submission (R1)> The related survey paper was not accepted at the main conference. This submission is self-contained and stands independently, providing methodology considerations for controlled user studies. We will remove references to the companion paper.
We will remove subsection numbers from Figure 1 (R4) and simplify dense sections for improved readability (R2), such as the Technology System Design and User Task and Interfaces sections. By simplifying these sections, we create space to include suggestions raised in previous points.
Meta-Review
Meta-review #1
- Your recommendation
Provisional Accept
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper presents a structured framework for designing empirical user studies in medical image computing and computer-assisted interventions, adapting established HCI concepts to promote more rigorous and clinically relevant evaluations.
All reviewers provided consistent positive evaluations (score = 4), highlighting the importance and timeliness of the topic, as well as the clarity and potential impact of the work on improving research practices within the MICCAI community.
The main limitations noted by the reviewers are that the contribution is largely conceptual, lacks a concrete case study or illustrative example, and has limited novelty due to reliance on existing HCI principles. Another issue is that the discussion of practical trade-offs and feasibility could be further strengthened, and a potential overlap with another submission should be clarified.
Despite these limitations, no major methodological flaws were identified, and the weaknesses primarily relate to presentation and completeness rather than fundamental issues.
Based on the consistent positive evaluations and the relevance of the contribution, a provisional accept is recommended.
The authors are encouraged to include a concrete example, clarify novelty, discuss practical considerations, and address the potential overlap in the final version.