List of Papers Browse by Subject Areas Author List
Abstract
In medical imaging, particularly for lesion segmentation and localization, leveraging weakly supervised learning can significantly reduce the annotation burden. However, the challenge often lies in finding the right balance between annotation time and accuracy. In this study, we address this issue by introducing a balanced approach that uses a multi-annotation pattern. We propose a plug-and-play, label-informed dynamic framework, referred to as MALFOY, designed to efficiently learn from various types of annotations, such as masks, bounding boxes, points, etc. During the training phase, the model is informed of the types or styles of annotations through label weights, which are then linked to the losses for different annotations. Upon completion of training, the optimal segmentation or detection is determined by searching for the best label weights on training records or validation set, guided by specific objective values. Our experiments span two distinct datasets: a polyp dataset comprising 3,515 2D colonoscopy images and a breast cancer dataset containing 1,315 3D DCE-MRI images. The results demonstrate that our proposed model is capable of (1) maximizing the use of diverse annotations through label weights, (2) effectively integrating knowledge from various sources by identifying the optimal weights, (3) performing robustly across both 2D and 3D images while reducing the reliance on precise mask annotations, and (4) can be easily extended for multi-expert annotations. Our code is publicly available.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/2169_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
https://github.com/fiy2W/MALFOY
Link to the Dataset(s)
PolypGen dataset: https://github.com/DebeshJha/PolypGen
SUN dataset: http://amed8k.sundatabase.org/
CVC-ClinicDB: https://www.kaggle.com/datasets/balraj98/cvcclinicdb
Etis-Larib dataset: https://www.kaggle.com/datasets/nguyenvoquocduong/etis-laribpolypdb
DUKE-BREAST-CANCER-MRI dataset: https://www.cancerimagingarchive.net/collection/duke-breast-cancer-mri/
BibTex
@InProceedings{HanLuy_MultiAnnotation_MICCAI2026,
author = { Han, Luyi AND Cai, Lishan AND Tan, Tao AND Zhang, Tianyu AND Gao, Yuan AND Wang, Xin AND Lu, Chunyao AND Liang, Xinglong AND Portaluri, Antonio AND Pinker-Domenig, Katja AND Sun, Yue AND Teuwen, Jonas AND Beets-Tan, Regina AND Benson, Sean AND Mann, Ritse},
title = { { Multi-Annotation Adaption: A Label-Informed Dynamic Framework for Medical Segmentation and Localization } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16881},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes to localize or segment medical images using multiple annotations together. Two kinds of searching strategies are utilized to find the optimal weight hyperparameters. Experimental results on different datasets demonstrate the effectiveness of the proposed method.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The major strengths of this paper include: (1) Weak annotations in the field of segmentation are classification tasks. Supervising the segmentation model with multiple annotations at the same time should be beneficial for producing promising segmentation masks. The idea of this work makes sense. (2) The experiments are detailed and sufficient to demonstrate the effectiveness of the proposed method. Particularly, the training convergence is also clearly presented.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The major weaknesses of this paper include: (1) The motivation is not very clearly introduced or discussed. It would be better to discuss why training with multiple annotations is beneficial and how it influences segmentation. The technologies and solutations in each steps are not very novel. (2) More visualization and analysis of segmentation for single and multiple annotations could be provided. (3) Multiple kinds of weak annotations are commonly utilized in the field of computer vision. The proposed method could also be used in other settings, but are there any results to verify its generalization?
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposes a framework for training segmentation with multiple kinds of annotations together. The experiments are detailed and verify the effectiveness of the proposed method.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This paper proposes MALFOY, a multi-annotation label-informed dynamic framework for medical segmentation and localization. The method is designed to jointly leverage heterogeneous annotation types, including precise masks, bounding boxes, points, inner and outer masks, and noisy masks. Its core idea is to inject annotation-type information into the segmentation model through label weights, which modulate the training loss and are also used to condition a dynamic network based on HyperConv layers.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The paper addresses a practically important problem in medical imaging: how to effectively combine multiple annotation types to reduce annotation burden while preserving segmentation and localization performance. 2.The empirical study is reasonably broad, covering both 2D and 3D settings, annotation-ratio experiments, weight-search analysis, and an extension to multi-expert uncertainty modeling, which strengthens the practical relevance of the work.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.Although the framework is useful, the methodological novelty appears moderate, since the overall design mainly combines dynamic weighting, dynamic convolution, and test-time weight search rather than introducing a fundamentally new learning principle. 2.The best-weight searching procedure is important to the reported gains, but it also adds extra complexity at inference time and raises questions about practical deployment efficiency, fairness, and reproducibility. 3.The experimental comparisons are limited, as they focus mostly on weakly supervised baselines and do not sufficiently compare against stronger multi-annotation learning, uncertainty-aware, or more modern segmentation approaches. 4.The paper would be stronger with a deeper analysis of why different annotation types complement each other, how the learned label weights behave in practice, and what the model is actually learning from heterogeneous supervision beyond the reported quantitative improvements.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission does not provide sufficient information for reproducibility.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I gave this paper a weak accept because it tackles a meaningful and practical problem and presents a reasonably effective framework with solid empirical improvements across both 2D and 3D datasets. The annotation-burden analysis and the extension to multi-expert supervision are also valuable. However, the methodological novelty is moderate, and a notable part of the performance gain depends on an additional weight-searching procedure at evaluation time. The paper would be stronger with deeper analysis, broader comparisons, and a clearer discussion of deployment efficiency and fairness.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
I appreciate the authors’ clarifications in the rebuttal. The response better motivates the clinical setting of heterogeneous annotations and explains how label weights serve as an interface between different annotation types and the segmentation model.
However, some concerns remain only partially resolved. The best-weight search remains an important component of the reported gains and still raises some questions about fairness, since it is unclear whether comparable objective-specific tuning is available to the baselines. The comparison to stronger modern segmentation or multi-annotation learning methods also remains limited, and the analysis of what the model learns from heterogeneous supervision is still relatively shallow.
Overall, despite these limitations, I believe the paper addresses a meaningful and practical problem, presents a coherent framework, and provides reasonably solid empirical evidence across 2D, 3D, and multi-expert settings. The remaining issues are important but can likely be addressed in revision rather than requiring rejection.
Review #3
- Please describe the contribution of the paper
This paper proposes MALFOY, a framework for dynamic aggregation and learning from a wide variety of medical image segmentation annotations, obtained at different annotation granularities (e.g., precise masks, bounding boxes, point annotations). MALFOY dynamically assigns (searched) weights to different annotations which are then used to weight losses used for the final optimization.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The problem setup of working with annotations-of-varying-granularity is quite interesting and potentially representative of real-world clinical settings. 2.The Searching-R approach of using a regressor to learn the relationship between loss values and label weights appears to be novel.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.While the concept of predicting losses (Eqn. 4) seems interesting, the authors do not explain why it is expected that the relationship between losses and label weights must account for subject ID, and this is also related to 2(a) below.
2.The paper is quite unclear in parts, with key details and visualizations missing to understand the method. For example: (a) How is the training data for Searching-R, i.e., \mathcal{R} in Eqn. 4 actually generated? (b) What is the scale of this dataset? How long does it take to train a random forest (RF) regressor on it? There are no details whatsoever about the RF implementation. (c) How good is this RF regressor? The paper is missing a quantitative evaluation of it. This is extremely important because a poorly trained regressor will lead to poorly optimized label weights.
3.There are no visualizations of different types of annotations, so (a) it is not clear what inner mask and outer mask look like for multi-class (C classes; Sec. 2.1) segmentation. (b) how are noisy masks obtained from precise masks? What kind of noise is added?
4.About the “label-informed adaptive learning loss” (Eqn. 3): (a) what loss is used for L_seg? Is it DSC? Is it CE? Not specified. (b) most importantly, how is Eqn. 3 calculated when Y_i is a bbox or point annotation, i.e., Y_2, Y_3? Because if we imagine L_seg is a segmentation loss, how is it computed against bbox/point “ground truth” annotation where there are no pixelwise ground truth labels?
5.It is unclear what the scale of compute required for Searching-{V, R} is. What do 100 and 10000 search trials actually amount to in real world CPU wall time? Because that information significantly affects the applicability of the proposed method.
6.From how it is described, Searching-V is not really a new “searching method” as the authors claim - it is just validation set-based selection, which is virtually ubiquitous in deep learning.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed method attempts to address an interesting challenge - can we leverage different qualities of annotations for medical image segmentation? But there are missing details about how the method is actually implemented: how is the dataset of Eqn. 4 created, how good the regressor is, what loss is used in L_seg, how is Eqn. 3 optimized when Y_i is not a dense label.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
I thank the authors for providing implementation details in their response. However, these missing technical and implementation would not all be possible to be accommodated in the revision, given the limited space available for revision, and without these details, the paper is hard to follow.
Moreover, despite these clarifications, a few things are unclear to me:
(a) the authors say “a point is converted to a circular mask” -> what does this mean? A point is a point mask with 1 pixel, not a circle. Do the authors dilate the point to make it a multi-pixel mask?
(b) Respectfully, I still do not see how the Searching-V is a new searching method. It is still validation set-based selection, but by proxy.
(c) Related to a comment by R1 about the motivation not being “very clearly introduced or discussed”, I would agree with this, especially because the authors do not clarify why, in scenarios where seg masks are available, localization AKA bounding box detection is a task to solve, and why the largest rectangle that fits the seg is not the localization.
Author Feedback
We thank the reviewers for their constructive comments.
R#1/#2 1.Motivation and novelty Our motivation is the clinically common setting where annotations of different granularities coexist, including precise masks, boxes, points, noisy masks, and multi-expert annotations. These annotations provide complementary supervision: masks offer boundary cues, boxes provide localization constraints, points indicate lesion presence, and noisy/inner/outer masks reflect uncertainty in lesion extent. Instead of treating weak labels as precise masks or training separate models for each annotation type, MALFOY uses label weights as a controllable interface between heterogeneous annotations and the segmentation model. Specifically, the weights jointly modulate the loss and condition the dynamic HyperConv layers, allowing the trained model to recommend different segmentation behaviors by searching for the optimal weight vector for a given objective.
2.Comparison and analysis The current experiments compare MALFOY with representative weakly supervised methods across single- and multiple-baseline settings. Table 2 shows consistent improvements in both segmentation and localization metrics on polyp and breast cancer datasets. Fig. 3 further analyzes different proportions of precise masks and bounding boxes, showing that MALFOY better benefits from additional weak annotations than the single-baseline setting. Fig. 4 visualizes searched label weights and shows that the best solution is generally not one-hot, indicating that different annotation types are complementary. Fig. 5 also demonstrates the extension to multi-expert uncertainty modeling.
3.Generalization MALFOY is evaluated on 2D colonoscopy, 3D breast DCE-MRI, and multi-expert prostate segmentation. These settings differ in modality, dimensionality, annotation type, and target anatomy.
R#3 1.Role of ID ID is only the sample index for Searching-R, not an input to the segmentation model or used during inference. It is included because the mapping from λ to loss is one-to-many: different cases can have different losses under the same λ. Thus, ID helps the RF regress case-specific λ-loss relations during weight search.
2.RF details We will add these details in the revision. “For each case in a batch, we record {epoch, ID, λ, loss} during training, producing approximately n_epochs × n_cases records with no extra training overhead. We use a sklearn RF regressor with 100 trees and default parameters. Records are split 8:2 for training and validation. Training takes 3.9 min on the polyp dataset and 1.6 min on the breast dataset. The RF achieves MAEs of 0.0025/0.0031 on training sets and 0.0066/0.0074 on validation sets for the two datasets, respectively, suggesting sufficient accuracy for ranking different λ values.”
3.Visualization of annotations (a) In this work, we only focus on binary masks for two datasets. For future extension, multi-class scenarios where inner/outer masks would be defined per class (e.g., on tumor label only, not organs). (b) The noisy mask is generated by applying a random B-spline deformable transformation to the precise mask, which introduces random shape and boundary perturbations while keeping the lesion roughly localized.
4.Loss (a) L_seg is defined in Eq.1, which comprises DSC and CE. (b) Each weak annotation is converted into a dense mask with the same resolution as the input image; e.g., a bbox is converted to a rectangular mask, and a point is converted to a circular mask. This unified approach allows identical loss computation across all annotation types, enabling label weighting to control annotation-specific learning.
5.Searching speed Searching-V takes around 156/77 min (polyp/breast) for 100 steps, and searching-R takes around 34/14 min for 10000 steps.
6.Novelty of Searching-V Validation set-based selection is mainly used to select the best checkpoint, whereas our searching-V selects the best λ for the given checkpoint.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Reviewer feedback was mixed. The authors are encouraged to address all concerns raised, with particular attention to the implementation details highlighted by Reviewer #3.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The authors omitted a lot of details, which should be added in the revised version to address the confusion of readers. Despite the points that need rewriting, the paper in its current vesion shows interesting results for the problem studied.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
I agree with the majority of the reviewers’ recommendation for acceptance. The remaining concerns raised after the rebuttal are primarily related to clarity and presentation. Despite these limitations, the paper addresses an interesting problem, presents a coherent framework, and provides reasonably solid empirical evidence across 2D, 3D, and multi-expert settings.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Reject
- Please justify your recommendation.
This submission received mixed scores during the review process. After reading carefully both the paper and authors rebuttal, I feel that not all the concerns have been satisfactorily addressed. In particular, responses concerning crucial implementation details, novelty and absence of more relevant baselines are unconvincing. For example, while I agree that [13] and [14] are relevant WSSS methods, these are outdated, and more recent approaches must be evaluated to better contextualize the actual benefits of the proposed approach.
Given these comments, I recommend Rejection of this work, and encourage authors to integrate the constructive feedback provided by the reviewers.