List of Papers Browse by Subject Areas Author List
Abstract
Adapter-tuned Segment Anything Models are promising for medical image segmentation, yet their backbone-heavy compute and adapter overhead hinder real-time deployment. Studying SAM-Med2D, we observe that structured masking harms Dice far more when applied to the frozen backbone than to adapters, suggesting strong asymmetry in redundancy. We propose RASP-SAM, a structured pruning framework that exploits this decoupling. It defines dependency-safe pruning units under attention, FFN, adapter, and residual couplings, ranks units with Orthogonal Residual Scoring based on the orthogonal residual of downstream gradients relative to a merged pretraining and downstream gradient subspace with cross-fitting, and restores accuracy using Dual-Stream Compensation that distills adapters and backbone in two stages under a small calibration budget. With only 20K calibration images from SA-1B and SA-Med2D-20M, RASP-SAM achieves near-lossless segmentation on SAM-Med2D while reducing encoder compute from 65.15G to 15.62G MACs and parameters from 270.1M to 63.4M at 76% sparsity, outperforming a transferred SlimSAM baseline at comparable cost. These results bridge adapter-tuned foundation models and resource-constrained clinical settings.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/1766_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
N/A
Link to the Dataset(s)
N/A
BibTex
@InProceedings{LuoXin_ResidualAware_MICCAI2026,
author = { Luo, Xin AND Wang, Chunjiang AND Zhou, Shaohua Kevin},
title = { { Residual-Aware Structured Pruning with Dual-Stream Alignment for Adapter-Tuned SAM } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16878},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
This paper proposes RASP-SAM, a structured pruning framework for adapter-tuned SAM in medical image segmentation. The key idea is to explicitly exploit the asymmetry between the frozen backbone and trainable adapters observed in SAM-Med2D, where pruning the backbone is much more harmful than pruning the adapters. Based on this observation, the paper introduces three main components: (1) dependency-aware pruning units that respect attention (2) Orthogonal Residual Scoring (ORS), (3) Dual-Stream Compensation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper addresses an important issue for medical foundation models: adapter tuned SAM variants can remain too heavy for real-time or resource constrained use, even when tuning is parameter efficient. The motivation is strengthened by the pilot study showing asymmetric sensitivity between backbone and adapters under structured masking.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The main comparison is against dense SAM-Med2D and a transferred SlimSAM baseline. The related work discusses FastSAM, MobileSAM, PQ-SAM, and other efficient SAM approaches, but these are not included in the main comparison table. Even if some are not directly plug-compatible with adapter tuned SAM, a stronger empirical discussion or additional baselines would improve the paper.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper addresses a meaningful and timely problem and does so with a reasonably novel, well-structured technical solution. it explicitly models the structural asymmetry between the backbone and adapters in adapter-tuned SAM, and introduces a coherent framework around this observation: dependency-safe pruning, ORS ranking, and dual-stream compensation. The empirical results on SAM-Med2D are convincing within that setting, showing substantial compression and real speedups with relatively small performance drops, and the ablation studies are useful in supporting the design choices.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
N/A
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #2
- Please describe the contribution of the paper
This work proposes a structured pruning method for SAM, aiming to reduce the model size while preserving its performance as much as possible. This is particularly valuable for resource-constrained medical scenarios. The proposed method is evaluated on two medical image segmentation datasets and outperforms baselines such as SlimSAM.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The Inference Speed Analysis is very thorough and effectively demonstrates the practical value of the proposed pruning method.
2.The study is highly meaningful for resource-constrained medical scenarios or applications with strict real-time requirements.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
1.While the research itself is valuable, the experimental comparison is insufficient. In Table 1, only SlimSAM is included as a baseline. Considering that there have been numerous works on structured pruning over the past decade, using only one comparison method is clearly not enough to demonstrate the superiority of the proposed approach.
2.The discussion of recent pruning methods is insufficient. The related work mainly includes studies prior to 2023, and some are not even specifically designed for SAM. If there are more recent pruning methods for SAM, they should be included. If not, the authors should at least discuss the most relevant Slim-series pruning methods in greater depth.
3.The comparison in Table 5 only partially supports the effectiveness of ORS. Why can ORS serve as an importance ranking criterion? What exactly does ORS measure? The theoretical justification and experimental evidence provided are insufficient. It appears more like an empirical design, which may limit its generalizability.
4.Many abbreviations and symbols are not clearly defined. For example, in the method section, the symbols in Figure 3 are not well aligned with the textual description. In the experiments, terms such as 1pt, 3pt, 5pt are used without explanation (what does “pt” refer to?) Additionally, configurations such as mean latency and TPS under different settings are not clearly presented. Overall, this negatively affects the reviewing experience.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The main factors influencing my score are the practical value of the proposed method and the thorough inference efficiency analysis, which demonstrate its potential in resource-constrained scenarios. However, the paper has notable limitations, including insufficient experimental comparisons, lack of discussion on recent related work, and unclear theoretical justification for key components. These issues weaken the overall contribution and rigor of the work.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
N/A
Review #3
- Please describe the contribution of the paper
The main contribution of this paper is a pruning-and-recovery framework for adapter-tuned SAM, instantiated on SAM-Med2D. The paper is motivated by an empirical observation that structured masking hurts the frozen backbone much more than the adapters, and then builds a three-part pipeline around this observation.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper addresses a relevant problem. Efficient deployment of SAM-style models is important for medical imaging, and compression for adapter-tuned SAM is less explored than either adaptation or generic SAM pruning.
The method is reasonably well motivated. The observation that the backbone is much more sensitive to pruning than the adapters gives a clear rationale for asymmetric pruning and for the two-stage recovery strategy.
The proposed framework is technically coherent. The dependency-aware pruning units, ORS scoring, and dual-stream compensation fit together well, rather than looking like unrelated tricks.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
The experimental validation is somewhat limited. Although the paper is framed around adapter-tuned SAM in general, all experiments are conducted on SAM-Med2D. It would be more convincing to test on other adapter-style SAM variants or direct comparisons with simpler pruning criteria under the same dependency grouping and the same compensation budget.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The work addresses an important problem for medical image segmentation and presents a technically coherent pruning framework for adapter-tuned SAM. The empirical results on SAM-Med2D are solid, with meaningful compression, useful ablations, and a reasonable accuracy-efficiency trade-off.
- Reviewer confidence
Somewhat confident (2)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
N/A
Author Feedback
We thank all reviewers for their constructive comments. Following the MICCAI policy, we address the concerns using the submitted results rather than adding new results. (R=Reviewer, W=Weakness)
R1, R2W1 and R2W2: Our goal is to structurally prune an already adapter-tuned SAM-Med2D while preserving its backbone–adapter decomposition. Therefore, we selected SlimSAM as the closest transferable pruning baseline, since it compresses SAM through pruning/distillation under a limited calibration budget, which best matches our setting. FastSAM is an architecture-redesign method, MobileSAM replaces the heavy SAM image encoder with a much smaller distilled encoder, and PQ-SAM is a quantization method that reduces numerical precision rather than structurally removing parameters. These methods are valuable efficient-SAM directions, but they optimize different efficiency axes and do not directly test adapter-aware structured pruning of an already adapter-tuned model. We also agree that recent pruning methods should be discussed more thoroughly. Recent ViT pruning methods such as MDP and VBP are relevant, but MDP is latency-constrained generic ViT/CNN pruning, while VBP mainly prunes MLP hidden neurons and does not cover attention, Q/K/V, embedding dimensions, or adapter-residual coupling. Thus, they are important related work but not direct baselines for dependency-safe pruning of adapter-tuned SAM. We will expand this discussion. Therefore, SlimSAM is the primary direct baseline in our setting; submitted Table 1 shows that RASP-SAM remains close to dense SAM-Med2D and outperforms transferred SlimSAM at comparable encoder cost.
R2W3: ORS aims to preserve non-substitutable downstream directions. It constructs a joint gradient subspace from upstream and downstream unit gradients, projects each unit’s downstream gradient onto this subspace, and uses the orthogonal residual as importance. A small residual means the unit’s downstream gradient is largely explainable by the joint subspace and can be better represented by the remaining structure; a large residual indicates downstream-relevant directions that the remaining structure cannot represent well, making the unit more important to preserve. ORS is used for backbone pruning in our submitted method. Submitted Table 5/Fig. 5 compare ORS with Magnitude, Hessian, and Taylor under the same B75 setting and 20K mixed calibration schedule, where ORS achieves the best Dice and faster convergence.
R2W4: We clarify that the terms mentioned by the reviewer are already defined in the submitted manuscript. Sec. 4 states that 1pt/3pt/5pt denote one/three/five point prompts. The Inference Speed Analysis paragraph before Table 2 defines Mean latency as average ms/image and TPS as throughput in images/s. We will move these definitions into the table captions for better self-containment. For Fig. 3, expansion/first layer refers to the first layer in FFN/adapter modules such as WF1/WC1, while reduction/second layer refers to the following WF2/WC2.The output dimension of the first layer and the input dimension of the second layer form the fine-grained local dependency. We will revise the caption to explicitly map colors/symbols to local and global dependency rules.
R3: We agree with the reviewer’s concern. We will therefore qualify our claims to avoid overgeneralizing beyond the validated setting. SAM-Med2D is still a representative testbed for RASP-SAM because it exposes the frozen-backbone/trainable-adapter decomposition and residual coupling that our method targets. Regarding the suggestion of comparing simpler pruning criteria under the same grouping and compensation budget, Table 5/Fig. 5 already show that Magnitude, Hessian, Taylor, and ORS are compared under the same B75 setting and 20K mixed calibration schedule. Following the reviewer’s suggestion, we will adjust the claim scope accordingly and explore broader adapter-style architectures in future work.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes a structured pruning framework, RASP-SAM, for adapter-tuned SAM in medical image segmentation. Reviewers generally agreed that the framework is technically coherent, with clear motivation from the observed backbone–adapter pruning asymmetry, meaningful compression results, useful ablations, and practical inference speed analysis. However, the rebuttal should address several concerns through clarification and appropriate qualification of claims, including the limited comparison mainly against SlimSAM, the need for deeper discussion of recent SAM/pruning baselines, clearer justification of what ORS measures and why it should generalize as an importance criterion, the limited validation on only SAM-Med2D despite broader claims about adapter-tuned SAMs, and several clarity issues around symbols, pruning settings, prompt terms.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper addresses an important deployment problem for adapter-tuned SAM and presents a coherent pruning framework with strong compression, practical speedup, and useful ablation evidence. Although the validation is limited mainly to SAM-Med2D and the baseline comparison is not exhaustive, the submitted evidence is sufficient to support the main contribution.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
All reviewers are positive about the work, and the clarifications presented in the rebuttal are convincing. I recommend acceptance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
I recommend acceptance. The paper addresses an important practical problem: compressing adapter-tuned SAM models for medical image segmentation while preserving performance and inference efficiency. The proposed RASP-SAM framework is technically coherent, motivated by the observed asymmetry between frozen backbone and trainable adapters, and supported by useful ablations and speed analysis.
The rebuttal reasonably addressed the main concerns. The authors clarified why SlimSAM is the most directly comparable pruning baseline in their specific adapter-tuned SAM setting, explained why other efficient-SAM approaches optimize different efficiency axes, clarified what ORS measures and how it differs from simpler pruning criteria, and agreed to narrow the claims to the validated SAM-Med2D setting. Reviewer #2 changed to accept after rebuttal, and the other available reviews are also positive.
