List of Papers Browse by Subject Areas Author List
Abstract
Medical image quality assessment (Med-IQA) is a prerequisite for clinical AI deployment, yet multimodal large language models (MLLMs) still fall substantially short of human experts, particularly when required to provide descriptive assessments with clinical reasoning beyond simple quality scores. However, improving them is hindered by the high cost of acquiring descriptive annotations and by the inability of one-time data collection to adapt to the model’s evolving weaknesses. To address these challenges, we propose MedQ-Engine, a closed-loop data engine that iteratively evaluates the model to discover failure prototypes via data-driven clustering, explores a million-scale image pool using these prototypes as retrieval anchors with progressive human-in-the-loop annotation, and evolves through quality-assured fine-tuning, forming a self-improving cycle. Models are evaluated on complementary perception and description tasks. An entropy-guided routing mechanism triages annotations to minimize labeling cost. Exp
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1141_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
MedQ-Engine Dataset: https://huggingface.co/datasets/jiyaoliufd/MedQ-Engine
BibTex
@InProceedings{LiuJiy_MedQEngine_MICCAI2026,
author = { Liu, Jiyao AND Ning, Junzhi AND Qu, Wanying AND Liu, Lihao AND Ma, Chenglong AND He, Junjun AND Xu, Ningsheng},
title = { { MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16879},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes MedQ-Engine, a closed-loop data engine that iteratively improves multimodal large language models (MLLMs) for medical image quality assessment (Med-IQA). The engine operates through three phases: (1) Evaluating, which clusters model failure cases into prototypes via data-driven analysis on a development set; (2) Exploring, which uses failure prototypes as retrieval anchors to expand training data from a million-scale image pool, combined with progressive human-in-the-loop annotation and entropy-guided routing to minimize expert cost; (3) Evolving, which fine-tunes the model on quality-assured data and re-enters evaluation, forming a self-improving cycle. With only 10K annotations, the method elevates an 8B-parameter model to surpass GPT-4o by over 13% and approach human expert performance across five medical imaging modalities.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The closed-loop evaluate-explore-evolve paradigm is well-designed and practically motivated. By iteratively identifying failure prototypes and using them as retrieval anchors, the engine targets the most impactful weaknesses rather than uniformly augmenting data, achieving over 4x sample efficiency compared to random sampling. 2.The progressive human-in-the-loop annotation strategy is highly practical. The entropy-guided routing mechanism reduces human review to only 18% of samples in later iterations while maintaining data quality, and the cold-start design with GPT-4o pre-annotation achieves a 10x speedup in per-sample review time. 3.Comprehensive experimental evaluation across five medical imaging modalities with thorough ablation studies (Table 2), scaling analysis (Figure 3), and annotation cost analysis (Table 3) convincingly demonstrates the effectiveness of each component. The results showing an 8B model outperforming all larger models (32B, 72B, closed-source) highlight the power of targeted data curation over brute-force scaling. 4.The method is general and not tied to a specific base model. It is demonstrated on both InternVL3-8B and Qwen2.5-VL-7B with consistent improvements, suggesting broad applicability of the evaluate-explore-evolve paradigm.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.In Table 1, the “Consistency” metric shows a slight decrease after MedQ-Engine optimization for both models (e. g. , InternVL3-8B: 1.858 -> 1.846). Although the drop is marginal, the authors should briefly discuss why this dimension does not improve alongside the others. 2.Figure 3 (a)(b) shows that performance on “No Degradation” images slightly decreases at larger data scales. It would be helpful to add a short discussion on whether the failure-driven sampling strategy introduces a mild bias toward degraded images at the expense of clean-image recognition.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper presents a novel and well-motivated closed-loop data engine that achieves impressive results: an 8B model surpassing GPT-4o and approaching human expert level with only 10K annotations. The method is general, practical, and thoroughly evaluated. The progressive human-in-the-loop annotation strategy is particularly valuable for reducing expert cost in medical AI. The identified weaknesses are minor.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I agree to accept this paper.
Review #2
- Please describe the contribution of the paper
This paper proposes MedQ-Engine, a closed-loop data engine designed to iteratively improve Multimodal Large Language Models (MLLMs) for Medical Image Quality Assessment (Med-IQA). The framework addresses the high cost of expert descriptive annotations and the static nature of traditional datasets by implementing a three-phase “evaluate-explore-evolve” cycle. The proposed method was evaluated across five medical imaging modalities.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Medical image quality assessment (Med-IQA) is a clinically relevant topic.
- Fine-tuning open-source LLMs with samples selected by the proposed pipeline improves Med-IQA performance.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Unclear, confusing method description. 1a) “We evaluate Mθ on Ddev R times across perception and description tasks spanning multiple modalities. A sample is identified as a failure case and added to the failure pool B when its error rate across R runs exceeds a threshold γ, indicating a persistent weakness rather than stochastic variance.” How do you define errors for description tasks? 1b) Eqn. (1): The vector e is not used afterward. 1c) Below Eqn. (2): “Adaptive sampling weights w_k ∝ e^α_k …” What is e^α_k? Where is w_k used? In addition, being proportional is not specific enough to define w_k. 1d) tau_H and tao_ann are not defined. 1e) “… subsequent iterations applying entropy-guided selective review.” What is entropy-guided selective review? 1f) How is D_dev constructed?
2.Use of GPT-4o for reference annotation, without any alternative options, is a concern. 2a) GPT-4o is a powerful proprietary LLM. It is unclear whether the entire pipeline for this work relies heavily on it. 2b) As the authors described, the data scale of this work is “approximately 1 million images”. The cost of employing GPT-4o for annotation at this scale should be disclosed to boost reproducible research.
3.Experiments. 3a) “totaling approximately 1 million images”: What are the distributions of the images across modalities? 3b) “For Description, we employ four expert-evaluated dimensions …” How many samples are manually evaluated? What are the max scores? 3c) Table 1: Are all compared methods NOT fine-tuned / post-trained for Med-IQA? In fact, the comparison should be with other training data selection methods, or at least with models fine-tuned with the 2K fixed samples in D_dev. In conclusion, the comparison is not meaningful or fair. 3d) “3.4 Data Scaling Analysis”: What are the definitions of mild and severe degradations?
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The insufficient description of the methodology, exclusive reliance on GPT-4o, and the improper experimental comparison concern me.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
I appreciate the authors’ detailed responses for clarification. However, I still have two major concerns: 1.There is too much confusion that the authors tried to clarify in the rebuttal. It is still difficult to fully clarify them, given the limited characters and the large amount of missing information in the original submission. 2.There is a lack of comparison with existing data-selection methods, which have been a major focus in the active learning field.
Review #3
- Please describe the contribution of the paper
This paper proposes MedQ-Engine, a closed-loop data engine for medical image quality assessment, which iteratively enhances MLLMs through an evaluate–explore–evolve pipeline with failure prototype clustering and entropy-guided human-in-the-loop annotation to maximize sample efficiency and reduce labeling cost.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.Introduce a data-driven failure discovery mechanism with error-weighted adaptive sampling, paired with a progressive human-in-the-loop annotation paradigm guided by trajectory-level entropy. 2.Provide a general data-efficient paradigm for adapting MLLMs in expert-scarce domains.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.The paper does not report the number of human experts, their backgrounds, inter-annotator agreement, or disagreement resolution. Adding these details will significantly improve the clinical rigor, reproducibility, and reliability of the work.
2.The paper does not provide sufficient details to verify data leakage prevention. “Disjoint patient splitting” is claimed but not justified with concrete statistics, and potential overlap between pre-trained MLLM training data and the proposed benchmark is not addressed.
3.The VQA data is aggregated from existing benchmarks, yet no cross-dataset annotation consistency verification is performed. Divergent criteria (e.g., artifact definition, severity judgment, description granularity) can introduce annotation noise.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Despite the minor weaknesses mentioned above, the paper’s systematic closed‑loop data engine for medical image quality assessment presents a novel, effective, and clinically valuable contribution that significantly advances MLLM performance in this critical domain.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have addressed my questions, so I lean to accept.
Author Feedback
Thank you for the careful reviews. We address each point below.
Reviewer #2 1) Consistency measures coherence between the model’s reasoning process and its final conclusion. Baseline scores are near the ceiling (InternVL3-8B: 1.858/2.0), leaving little room for improvement. The decrease is on the order of 1e-2, negligible relative to gains across all other dimensions. 2) The drop on No-Degradation cases is small. Figure 3(a)(b) shows a non-monotonic trend: performance dips slightly at smaller scales before recovering. This briefly perturbs the model’s prior for artifact-free images. As scale increases, this prior is restored, while gains on mild and severe degradations remain substantially large。
Reviewer #4 1a) Perception failures are defined by answer incorrectness. Description failures are defined by total score <5.1b/1c) Eq. (1) computes capability-wise error rates driving adaptive sampling weights in Phase 2 (w_k ∝ e_k^alpha, normalized as w_k = e_k^alpha / sum_j e_j^alpha, alpha=1). 1d) tau_H (=0.8). tau_ann: for perception, delta^ann is exact-match (tau_ann=1); for description, it reflects pass/fail agreement under MedQ-Bench scoring with threshold 5.1e) This is described in Sec. 2.3 under Self-evolution (t>0): trajectory-level entropy H_traj serves as the condition for accepting annotations. 1f) D_dev is from the MedQ-Bench develop split, independent of the test split, with deduplication and resampling for balanced modality/question-type coverage. 2a-2b) GPT-4o is used only for cold-start annotation, reference annotation, and description judging. To reduce cost, GPT-4o annotation is applied only to prototype-retrieved images. GPT-4o can be replaced by capable open-source models (e.g., Qwen3-235B) with comparable results. 3a) The ~1M image pool is balanced across five modalities (~200K each). 3b) Description scoring uses four dimensions (Completeness, Preciseness, Consistency, Quality Accuracy), each up to 2 points (total max score is 8). GPT-4o judging achieves high agreement with expert judging per MedQ-Bench, so we did not conduct an additional human evaluation study. 3c) Table 1 positions our optimized models against current MLLMs. Method-side validation is provided by Table 2 ablations and the random-sampling baselines in Figure 3(c)(f), which consistently underperform our approach at all data scales. Fine-tuning on a fixed 2K D_dev subset yields results close to Ours-2K (Qwen2.5-VL: 73.0% perception, 4.79 description). But this is limited by scale, and performance improves further when we expand the scale using our method. 3d) Mild/severe degradation splits follow MedQ-Bench definitions.
Reviewer #5 1) Description evaluation follows the MedQ-Bench protocol; MedQ-Bench reported high GPT-4o/expert agreement. We did not perform a separate agreement study. 2) MedQ-Engine is a training data engine; MedQ-Bench is an existing, independently curated benchmark. Training data come entirely from public datasets. U, D_dev, and D_test are from disjoint patient cohorts, and perceptual-hash deduplication confirmed zero image overlap across all splits. 3) Dedicated cross-dataset Med-IQA benchmarks are currently scarce. MedQ-Bench itself aggregates images from multiple independent sources across five modalities, providing inherent cross-source diversity. Our training pool is similarly multi-source and disjoint from MedQ-Bench. All conclusions are based on relative gains under the unified MedQ-Bench protocol.
We hope the above responses have adequately addressed the raised concerns.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The scores are generally good but mixed. After reading it, I think the paper concerns especially from #R4 are fixable. The rebuttal should focus on clarifying the methodological details raised by R4, and authors are still encouraged to address the concerns by R2 and R5 even though they have positive scores.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
MedQ-Engine is promising and received two accept recommendations. One rejection argues that a large amount of essential information was missing from the submitted manuscript, requiring rebuttal clarification, and the comparison lacks established active-learning/data-selection baselines. Given the paper’s central claim as a data engine, this weakens the evidence that the proposed selection strategy is meaningfully superior. Yet, I also follow the other two positive reviewers and give a borderline acceptance based on the potential and merits.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
After carefully considering the reviews and rebuttal, I recommend rejecting the paper. While the work presents promising empirical results and a potentially useful framework for reducing annotation burden, significant concerns remain regarding the clarity of the methodology, the adequacy of experimental comparisons, and the degree of novelty relative to existing data-selection and active-learning approaches. In particular, several key aspects of the method required substantial clarification during rebuttal, yet important questions remain unresolved. Given these concerns, I do not believe the paper currently provides sufficient evidence to support its claimed contributions for acceptance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This article presents a closed-loop data engine for medical image quality assessment, which is a clinically valuable contribution. While R4 is negative to the paper, the overall strengths outweigh the identified weaknesses. Therefore, AC recommends acceptance. The authors are encouraged to further improve the clarity of the final paper.