Abstract

Accurate 3D medical segmentation requires models that are both robust to anatomical variability and efficient for deployment. While knowledge distillation (KD) supports model compression, conventional voxel-wise or scalar-relation methods can be sensitive to architectural mismatch and background-dominated 3D volumes. We propose Displacement Preserving Relational Distillation (DPRD), which aligns normalized pairwise displacement vectors among ROI-pooled case-level embeddings to provide scale-invariant local relational supervision. Integrated into nnU-Net, DPRD outperforms representative KD baselines on ISLES 2022 and AMOS 2022. On AMOS, DPRD achieves 85.46% Dice and 11.72 mm HD95, slightly exceeding the MedNeXt teacher in Dice while the MobileUNetV3 student uses only ~5% of its parameters and ~3% of its FLOPs. Code: https://github.com/ClinicaAlpha/DPRD-3D-MedSeg

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/6226_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/ClinicaAlpha/DPRD-3D-MedSeg

Link to the Dataset(s)

ISLES 2022 dataset: https://isles22.grand-challenge.org/ AMOS 2022 dataset: https://era-ai-biomed.github.io/amos/dataset.html

BibTex

@InProceedings{DinZhi_Displacement_MICCAI2026,
        author = { Ding, Zhicheng AND Chu, Xinyu AND Choi, Jung Im AND Tian, Qing AND Shi, Tianyu AND Jiang, Xiaoqian AND Zhu, Lijing AND Lan, Qizhen},
        title = { { Displacement Preserving Relational Distillation for Robust Medical Segmentation } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16884},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    This paper proposes Displacement-Preserving Relational Distillation (DPRD) for 3D medical image segmentation. Instead of matching voxel-wise activations or scalar relational quantities such as pairwise distances or angles, DPRD first projects teacher and student features from each encoder stage into a shared 128-channel space, then applies ROI-aware density-normalized pooling to obtain case-level embeddings, and finally distils the pairwise displacement vectors among samples within a mini-batch. Separate normalization in teacher and student spaces is used to reduce scale mismatch across heterogeneous architectures. Because the method operates on compact case-level embeddings rather than dense voxel-wise relations, it is computationally more practical for 3D segmentation. Experiments on ISLES 2022 and AMOS 2022 show improvements over several knowledge distillation baselines.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    1.The paper addresses a relevant practical challenge in 3D medical segmentation, namely efficient distillation across heterogeneous teacher-student architectures. A notable aspect of the method is that it goes beyond standard feature alignment by distilling pairwise displacement vectors after mapping both networks into a shared latent space, thereby encouraging the student to preserve the teacher’s relational geometry in a way that is more expressive than scalar distance/angle matching and still efficient for 3D settings. 2.The proposed ROI-aware masking and density-normalized pooling are intuitive and well-motivated. They provide a reasonable way to suppress background-dominated signals, which is especially relevant in class-imbalanced 3D medical segmentation tasks. 3.The experimental section is reasonably comprehensive. The paper compares against several representative KD baselines and includes ablation studies on the main components. Qualitative visualizations are also helpful.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    1.The paper provides indirect evidence that displacement-based distillation is effective, mainly through ablation and downstream segmentation improvements. However, it does not yet directly demonstrate that the learned feature embeddings are particularly well suited to displacement-based relational modelling. More explicit feature-space analysis, such as visualization or quantitative evaluation of displacement alignment between teacher and student embeddings, would make this claim much more convincing. 2.The experimental section lacks statistical significance analysis. This is particularly important when some metric differences are relatively small. 3.Although the experiments are conducted within the nnU-Net framework, the paper still lacks sufficient implementation details to ensure reproducibility. The authors are encouraged to provide more detailed settings and, ideally, make the code publicly available. 4.Some presentation issues remain in writing, organization, and figure clarity. See comments.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    1.It would be better to introduce the mask M directly when it first appears in Equation (6), rather than only in the following paragraph. 2.On page 5, after Equation (8), the word “where” should not be indented, in order to stay consistent with the rest of the manuscript. . 3.Please reconsider the presentation order of RAFM and DPRA. From the current description, it seems that RAFM is applied before DPRA, so the introduction order may need to be adjusted accordingly. 4.In the experimental settings, only the training and validation sets are described, without a separate test set. Which set was used for parameter tuning? How did the authors ensure that the validation set was not overused during hyperparameter selection? 5.Fig. 2 is helpful for illustrating the function of the proposed method, but including a comparable figure from another method would make the comparison more convincing. 6.In the caption of Fig. 3, it would be helpful to explicitly indicate which colours correspond to the “gallbladder and postcava”, so that readers can locate them more easily. In addition, the current Fig. 3 does not clearly demonstrate what specific advantage the proposed method has over the other methods in the region highlighted by the green circle. 7.For Baseline-MobileUNetV3 and Baseline-PlainConvUNet, were these models trained directly on the training set in a fully supervised manner, without distillation? This should be clarified to avoid possible misunderstanding. 8.Is there any concern about using different student models for different datasets, like PlainConvUNet for ISLES 2022 and MobileUNetV3 for AMOS 2022 (CT)? It would be helpful if the authors could justify this choice and discuss whether it affects the fairness of the comparison. 9.Since one of the main motivations of RAFM is to alleviate class imbalance, it may be helpful to report class-wise Dice scores in the ablation study to better demonstrate its effectiveness.

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    Although there are still some weaknesses in the analysis, experimental clarity, and presentation, the method is technically reasonable and practically relevant. The comparisons with multiple baselines, together with the ablation studies, provide reasonably convincing evidence of its effectiveness for 3D medical segmentation. Overall, I lean toward acceptance, as most of these issues appear to be addressable through relatively minor revisions.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The authors have reasonably addressed the batch-size and normalization concerns. As R2 also noted, the heterogeneous distillation design is meaningful and practically useful, especially given the large reduction in computation. While the case-level displacement geometry still needs stronger theoretical or direct empirical support, the proposed method offers an effective solution for heterogeneous model distillation and demonstrates strong empirical performance, though some of its underlying assumptions and theoretical motivations remain open to discussion. Therefore, I support acceptance.



Review #2

  • Please describe the contribution of the paper

    The authors propose a method for knowledge destillation for medical segmentation networks, which aims to preserve the geometry of the manifold of the latent space of the teacher network when transfering the knowledge to the student network. The geometry of the latent space is captured using pair-wise difference of latent per-case feature vectors derived from pooled feature maps corresponding to foreground voxels, and the student network is trained to preserve the trajactories and distances in the latent space.

    The method was evaluated on a public MRI stroke lesion dataset, and a public CT abdominal organ segmentation dataset, with the results showing that preservation of the geometry of the latent space during knowledge destillation yields more accurate student models than competing knowledge destillation methods and may even yield to models that generalize better than or on par with the teacher model, while requiring orders of magnitude fewer parameters and GFLOPs.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The authors address an important practical consideration when training neural networks for image segmentation: how can I create smaller models that can be used under tight hardware constraints, while preserving the accuracy of large state-of-the-art models. This may further help with the adoption of deep learning and may bring current state-of-the-art accuracy closer to the patient onto devices, which are usually not equipped with the compute resources required to infer large transformer-based architectures.

    The authors make several contributions, which together yield in a strong knowledge destillation approach, starting from how to create an efficient per-case representation that is not dominated by background voxels, to how can we capture the geometry of the latent space and define similarity such that those measures can easily be integrated in existing deep learning frameworks simply as an additional differentiable loss function.

    The evaluation is well designed and suitable to show the advantage of knowledge distillation compared to several reasonable alternatives. That is, the student network is always compared to the teacher model, to see how large the gap in knowledge transfer still is after distillation. It is also compared against a “student” network, which was trained from scratch, showing that knowledge distillation (independent of which method is used for KD), can yield better models. Last but not least, the proposed KD approach was compared against state-of-the-art methods showing the better generalization capabilities of the proposed method. The evaluation was done on an MR and on a CT dataset, and a pathology and anatomy segmentation task, showing the generalization potential of the method to many different domains.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    Although the method can be used to train networks that require fewer hardware resources to infer, it is not clear from the paper what the impact on the training is and how the method is implemented in practice. Can the latent per-case vectors be pre-computed or do they need to be inferred during the training loop? Since the latent features are not used directly but fed through a 1x1x1 convolutional layer before pooling, I assume that they cannot be pre-computed and that those projection layers need to be trained jointly during knowledge distillation. The teacher network is frozen so the inputs to the projection layer can be pre-computed, but storing all feature maps per case is likely not feasible, so my assumption is that the feature maps need to be inferred during training, which significantly adds to the ressource requirements for training, compared to other distillation methods such as logit-based approaches, where the logits can be pre-computed.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not mention open access to source code or data, but provides a clear and detailed description of the algorithm to ensure reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper addresses an important problem to drive adoption of deep learning-based methods for medical imaging. Not all algorithms can run in the cloud on endless resources and bringing algorithms closer to where they are applied and used is crucial in many domains. The paper addresses this issue and addresses it well. The proposed method is sound, well motivated, and convincingly evaluated.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Accept

  • [Post rebuttal] Please justify your final decision from above.

    The main concers where addressed. Training time is comparable to logit-based training, because in both cases the feature space needs to be inferred during training, because data augmentation makes pre-computation infeasible in both settings.

    The concern of relatively small batch size was addressed. It is a stochastic training method, and the global structure of the feature space is only revealed after iterating over many epochs.



Review #3

  • Please describe the contribution of the paper

    This paper proposes a knowledge distillation method for 3D medical image segmentation called DPRD.This paper evaluates DPRD on two publicly available benchmarks: ISLES 2022 and AMOS 2022 . Using MedNeXt as the teacher model and PlainConvUNet and MobileUNetV3 as lightweight students, it compares DPRD with distillation methods such as Logits KD, FitNet, RKD, and CIRKD. Experimental results show that DPRD consistently outperforms all baselines on metrics such as Dice, HD95, and NSD, and slightly surpasses the teacher model on AMOS, while requiring only about 5% and 3% of the parameters and computational cost of the teacher model, respectively.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    The method consists of two core modules: Displacement-Preserving Relational Alignment (DPRA), which computes feature vector differences between pairs of samples within a batch and matches the displacement vectors of the teacher and student to transfer geometric structure. ROI-Aware Feature Masking (RAFM), which uses ground-truth segmentation labels to generate anatomical masks and performs density-normalized pooling on feature maps to obtain scale-invariant case-level embeddings. The distillation loss is applied to multiple stages of the encoder with different stage-wise weights.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    (1)In Section 2.1, the embeddings of the teacher and student are independently normalized per dimension, causing the displacement vectors of the teacher and student to not be aligned in the same linear space. The paper does not prove that this operation preserves the relative geometric structure of the manifold, nor does it validate the geometric meaning of the displacement matching loss. (2)The experimental batch size is only 2 or 4, generating only 1 or 6 displacement vectors per batch, providing extremely limited supervision signals. The paper does not analyze or verify whether such few vectors can effectively convey the structural information of a high-dimensional manifold, leaving the validity of the method in question. (3)RAFM relies on ground-truth labels to generate masks, limiting its applicability in few-label or label-free distillation scenarios. The paper does not explore the effect of replacing ground-truth masks with pseudo masks predicted by the teacher.

  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The submission does not provide sufficient information for reproducibility.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (2) Reject — should be rejected, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The major factors for rejection are: (1) the displacement vector matching relies on extremely small batch sizes (2 or 4), generating only 1–6 vectors per batch, which is insufficient to represent high-dimensional geometric relationships; (2) the ablation study fails to isolate the contribution of the Transformer bottleneck from the proposed distillation loss, leaving the source of performance gains unclear; and (3) the empirical improvements over baselines are marginal without statistical significance testing. Therefore, the methodological rigor is insufficient for acceptance.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    Reject

  • [Post rebuttal] Please justify your final decision from above.

    The rebuttal explains the training overhead more clearly, yet DPRD still requires online teacher forwarding during training, and the reported efficiency advantage mainly applies only to inference.

    The geometric claim is also weakened by the clarification. DPRD is now described as a scale-invariant local relational constraint rather than true manifold preservation, which makes the original claims about preserving teacher geometry somewhat overstated. The small-batch issue is partially addressed, but the evidence is still limited and does not fully justify stable relational distillation under general 3D segmentation settings.

    In addition, the reliance on ground-truth ROI masks during training limits applicability to fully supervised settings. The pseudo-mask extension is only discussed as future work, not validated. The added statistical tests are useful, but they do not resolve my broader concerns about the method’s conceptual strength, generality, and presentation.

    Overall, the rebuttal clarifies several implementation details, but it does not sufficiently address my concerns. I therefore maintain my negative recommendation.



Author Feedback

We thank the reviewers and meta-reviewer for their helpful comments. We summarize our responses to the main concerns below.

[R2 and Meta-reviewer: Training overhead] DPRD embeddings are computed online during distillation training. Each augmented patch is forwarded through the trainable student and the frozen teacher, while the 1 × 1 × 1 projection layers are optimized jointly with the student. Precomputing dense teacher features is not practical because training involves online patch sampling, data augmentation, and ROI masking. The extra cost of DPRD is negligible compared with standard online logit KD, since both are dominated by the teacher forward pass. On AMOS with batch size 4, online logit KD required about 40.3 GB GPU memory and 241 seconds per epoch, while DPRD required about 40.3 GB and 242 seconds per epoch. This overhead exists only during distillation training. At inference time, the teacher and projection layers are removed, so the final model has the same parameters and computational cost as the student trained without distillation.

[R3 and Meta-reviewer: Normalization, geometry, and small-batch relations] DPRD does not apply per-dimension normalization. At each stage, it uses one scalar relational scale for each stream and mini-batch, computed from the mean L2 norm of displacement vectors. This removes global activation-scale mismatch without whitening features or reweighting individual dimensions. After teacher and student features are projected into the same latent dimensionality, DPRD aligns local displacement directions and normalized distances. Thus, DPRD should be understood as a scale-invariant local relational constraint, not as a claim of global manifold isometry. We will clarify this point in the paper. Regarding the concern about small batch size, DPRD does not rely on a single mini-batch to provide dense relational structure. Across training, displacement vectors are repeatedly sampled from different cases, augmented patches, ROI regions, and encoder stages. The relational supervision therefore accumulates across iterations and epochs. Under matched training-sample budgets on AMOS, with batch size × epochs set to 2×300, 4×150, and 6×100, the Dice/NSD/HD95 scores were 81.31/79.31/15.63, 82.64/81.07/12.85, and 81.40/79.71/15.54, respectively. Increasing batch size did not monotonically improve performance, suggesting that DPRD benefits mainly from accumulated cross-iteration supervision rather than requiring dense within-batch relations.

[R3: Applicability under limited labels] RAFM uses ground-truth masks when labels are available because they provide more reliable anatomical guidance than predicted masks. In few-label or label-free settings, RAFM can instead use pseudo masks generated by the teacher, without changing the overall framework. We will clarify this applicability in the revised manuscript and discuss pseudo-mask-based RAFM as a future extension.

[R1: Evidence, significance, and presentation] To provide feature-level support, we have Fig. 2, which visualizes encoder-stage-3 feature heatmaps in axial, coronal, and sagittal views. These results show that DPRD guides the student to attend more accurately to critical lesion regions, supporting that DPRD guides the student toward task-relevant regions. Using the AMOS validation outputs underlying Table 2, paired Wilcoxon signed-rank tests show that DPRD significantly improves the MobileUNetV3 student across all metrics: Dice improves by 5.3 percentage points with p = 1.63 × 10⁻¹¹, NSD improves by 7.42 percentage points with p = 1.63 × 10⁻¹¹, and HD95 decreases by 7.02 mm with p = 5.56 × 10⁻⁷. We will further revise the manuscript to improve writing, organization, experimental clarity, and figure presentation. All code will be released to support reproducibility.




Meta-Review

Meta-review #1

  • Your recommendation

    Invite for Rebuttal

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    This work proposes displacement-preserving relational distillation for 3D medical segmentation, distilling pairwise displacement vectors between case-level embeddings rather than voxel-wise features. The ROI-aware density-normalized pooling is well motivated for class-imbalanced 3D tasks. R2 was strongly positive and recommended this for a best paper award. R3 gave 2 with specific technical objections.

    R3 raised two concerns worth addressing. First, with batch sizes of 2 or 4, the method generates only 1 or 6 displacement vectors per batch, and R3 questioned whether this provides enough supervision to capture high-dimensional manifold structure. Second, R3 noted that independently normalizing teacher and student embeddings means the displacement vectors are not in the same linear space, and the paper does not prove this preserves the intended geometric properties. R1, despite giving 4, also noted the lack of direct feature-space analysis to validate that displacements are actually aligned. R2’s only concern was about training resource requirements when the projection layers require teacher features to be inferred during training.

    For the rebuttal, please clearly address the following issues:

    1.address R3’s batch size concern and whether accumulating vectors across batches or epochs mitigates it; 2.justify or analyze the effect of independent normalization on geometric preservation; 3.clarify the training procedure and its computational overhead relative to logit-based distillation. R3’s concerns are specific and technical; if the authors can provide a convincing response, the paper is strong enough for acceptance given R2’s enthusiastic endorsement.

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The rebuttal settles the training-cost worry, because the extra cost is only at training and the small model runs as cheaply as a normal student, while the main result is already in the paper. It beats four named distillation methods on two datasets and nearly matches the teacher at about 3 to 5 percent of the size.

    Although I believe the paper is at the accept borderline due to its performance, I agree with R3 about the concerns. The Camera-ready must drop the overstated “geometry preserving” claim (if not the paper can’t be accepted due to overstatement, this will be checked in the camera ready version) and reword it as a local scale-invariant relational constraint.



Meta-review #2

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    While there are some controversial issues in this paper, especially some technical details that I am not fully familiar with, I am more concerned with the practical and empirical performance. In this regard, the paper shows good results in both quantitative and qualitative comparisons. For most of the issues, I think the authors effectively addressed them in the rebuttal, especially those from the meta-reviewer. Also, both Reviewer R1 and R2 still maintained their support for acceptance. Overall, I think it is a solid paper, and I am inclined to accept it.



Meta-review #3

  • After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.

    Accept

  • Please justify your recommendation.

    The paper proposes a relational distillation approach for 3D medical image segmentation which transfers teacher’s knowledge to the student while preserving local latent space structure. I feel the authors have done a convincing job with the rebuttal and the paper has enough merits to be accepted at MICCAI.



back to top