Abstract

Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose PolypSteer, a training-free activation-steering framework for endoscopic synthesis. PolypSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate PolypSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, PolypSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, PolypSteer achieves 75% dye removal against 20% (PnP) and 10% (h-Edit). On downstream polyp detection, augmenting with PolypSteer counterfactual pairs achieves ViT AUC of 0.9755 versus 0.9083 for quantity-matched re-prompting, confirming that counterfactual structure drives the gain.

Links to Paper and Supplementary Materials

Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4342_paper.pdf

SharedIt Link: Not yet available

SpringerLink (DOI): Not yet available

Supplementary Material: Not Submitted

Link to the Code Repository

https://github.com/UARK-AICV/PolypSteer

Link to the Dataset(s)

N/A

BibTex

@InProceedings{PhaTro_PolypSteer_MICCAI2026,
        author = { Pham, Trong Thang AND Nguyen, Loc AND Nguyen, Anh AND Nguyen, Hien V. AND Le, Ngan},
        title = { { PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering } },
        booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
        year = {2026},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 16892},
        month = {September},
        page = {pending}
}


Reviews

Review #1

  • Please describe the contribution of the paper

    The main contributions of this study are as follows: 1) The paper’s primary contribution is a training-free framework for generating counterfactual endoscopic images using only activation directions, without requiring a source image, mask annotation, or additional fine-tuning.

    2) The paper shows that pathology-specific directions in cross-attention activations can be leveraged to produce minimal-edit counterfactuals while preserving non-target anatomical and visual structure.

    3) The paper is also strengthened by its analysis of the method from three complementary perspectives: counterfactual generation quality, downstream detection utility, and disentanglement of clinically entangled attributes.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • The paper introduces a novel formulation of counterfactual generation that relies solely on activation directions. This approach is distinguished by its ability to operate without masks, source-image inversion, or other forms of auxiliary supervision required by prior research [1,2].

    • The proposed method effectively addresses the challenge of clinically entangled attributes through cosine-aligned steering within the model’s internal representation. By demonstrating that pathology-related factors can be selectively manipulated via cross-attention directions—without explicit disentangling labels or handcrafted controls—the work maintains structural integrity, making it both technically significant and highly relevant for medical imaging.

    [1] Sharma, Vanshali, et al. “Controlpolypnet: towards controlled colon polyp synthesis for improved polyp segmentation. “ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Qadir, Hemin Ali, Ilangko Balasingham, and Younghak Shin. “Simple U-net based synthetic polyp image generation: Polyp to negative and negative to polyp. “ Biomedical Signal Processing and Control 74 (2022): 103491.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • Limited Technical Analysis of the Steering Mechanism: The technical analysis of the cosine-similarity-based steering mechanism is relatively narrow. Specifically, the paper does not compare the proposed gating rule against alternative formulations, such as normalized cosine similarity, ungated dot-product steering, or soft gating functions, nor does it provide a quantitative evaluation of whether high-response tokens actually correspond to lesion regions. While the paper includes ablations over layer range, steering strength, and number of seeds, these experiments do not fully justify the specific design choice of the proposed gate. A more thorough analysis along these lines would make the methodological contribution more convincing.

    • Limited Breadth of Empirical Comparison: The empirical evaluation appears somewhat limited relative to the scope of the paper’s claims. Although the authors compare against re-prompting, PnP, and h-Edit, the paper would have been stronger if it had also included comparisons with mask-based counterfactual or controlled editing methods [1,2]. Even if such methods do not operate under the same source-free and annotation-free setting, they are still important reference baselines in medical image editing, and their inclusion would have provided a clearer picture of the practical advantages and trade-offs of the proposed approach within the broader literature.

    - In Table 1, the column header “CF?” looks slightly awkward and may be mistaken for a typo.
  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The paper proposes a novel training-free activation steering framework for counterfactual endoscopic image generation. Although the experimental setting is somewhat limited, the methodological novelty is strong enough to make the paper positively compelling.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #2

  • Please describe the contribution of the paper

    The paper introduces a way of generating counterfactuals without inversion methods often used in diffusion models to backtrack the trajectory and modify image contents. Explicitly they generate counterfactual pairs with only a “steered” concept being different in the two images. In essence it is not editing an image to generate counterfactuals but generating counterfactual pairs. To do this the authors make use of frozen DiT to generate concept vectors which are used to steer the inference away from the concept. The authors introduce these concept vectors to modify specific layers at each time-step in the DiT for steered inference.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
    • Creating a pair of counterfactual image pairs bypassing the need for inversion.
    • Guiding the inference with pathology vectors by updating different layers is a strong idea.
    • Good spatial interpretability offered
  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
    • There is a risk of domain generalization in this method since both kvasirv3 and hyperkvasir, the datasets used are similarly sourced, even removing near duplicate images doesn’t prevent the fact that same patient data could be present in training and in the testing set, possibly contaminating results.
    • An additional GI dataset, differently sourced or with different acquisition characteristics would further reinforce the effectiveness of the methodology.
    • In ablation the authors say that, “showing that layers 8–16 are the semantic formation zone”, however the selection of these layers seem more arbitrary, potentially providing references could benefit in the selection of these layers. This is concerning as the difference looks drastic in layer selection. Even when the layers 8-20 which include these “semantic formation” zone are selected the model seems to perform poorly.
  • Please rate the clarity and organization of this paper

    Satisfactory

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    The method to bypass inversion in counterfactuals by generating pairs through the use of pathology vectors is sound. Evaluations explore possible regions where the method is most powerful. I made the recommendation with these considerations in mind.

  • Reviewer confidence

    Confident but not absolutely certain (3)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Review #3

  • Please describe the contribution of the paper

    This paper proposes a method for steering medical image synthesis models, for the purpose of generative counterfactual examples for endoscopic images. It is based on the extraction of vectors of the cross-attention of a Diffusion Transformer (DiT). Positive and negative vectors are extracting, in a standard contrastive fashion, to establish a steering vector $v_t$, with which the model can be steered, and these vectors are subsequently used at inference time to create the counterfactual images.

  • Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

    I felt that the paper’s goal is clear, the method is straightforward and seems very interesting. I feel that the technical section could have been expanded a bit more to be clearer, as I explain below. The quantitative results seem good, although it is true that the visual results are a bit difficult to evaluate if one is not an expert (though this is normal). In any case, the impact of the method is clear in the results. The interpretability section is also good. While it is true that establishing a steering vector using positive and negative examples has already been explored in other works (in particular for vision-language models), this is acknowledged by the authors, and I feel that the contribution is strong enough here. The contrastive prompt design is clearly and straightforwardly presented.

  • Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.

    I would have liked a bit more clarity on exactly what the cross-attention (CA) features are ? Do you mean the output of the attention layer ? In a DiT, the cross-attention usually corresponds to the similarity between image tokens and text, or other conditioning, tokens, but the total number of tokens (image + text) is just referred to as the “tokens” I thought. So what is a cross-attention feature ? It does not correspond to the actual attention itself, if I understand correctly. I think it would help if you specified the size of this feature (the space in which it lives), even if this will probably require some further notation. This lack of clarification is quite visible when you first discuss $h_{l,t,n}$ and then $h_{l,t}$ without too much discussion. In particular, “where $h_{l,t}$ denotes the activation of any visual token at layer $l$, timestep $t$” is introduced much after $h_{l,t,n}$.

    p. 4 : “where $h_{l,t}$ denotes the activation of any visual token at layer $l$, timestep $t$”. Please detail this (as mentioned above).

    A weakness is a relatively small novelty in the steering vector creation, but I feel that the simplicity and impact of the method outbalances this.

    Another weakness is that the code is not released yet. The authors state that it will be released on acceptance, but this does not really guarantee anything. However, the method is simple enough to be re-implemented independently I feel.

    More detailled remarks:

    p. 4: “Because pathological changes are spatially focal” is this a typo ? p. 4: “following the mean-difference principle” what is this principle ? Not clear, and I do not think it is sufficient to just put a reference p. 4: “We score each token’s alignment with the pathology concept via $\cos(h_{l,t}, v_{l,t})$ …” Why does it make sense to take the dot product of an activation and the pathology vector ? If I understand correctly, you want to avoid modifying non target tokens, but why does the dot product do this ? Are you supposing that non target tokens live in an orthogonal space to $v_{l,t}$ ? If so, it would be good to have an explanation. p. 5: “A pre-trained oracle classifier” : what is this, what is the architecture ? More details please.

  • Please rate the clarity and organization of this paper

    Good

  • Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.

    The authors claimed to release the source code and/or dataset upon acceptance of the submission.

  • Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?

    N/A

  • Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html

    N/A

  • Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.

    (5) Accept — should be accepted, independent of rebuttal

  • Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

    I feel that the paper has a good balance between impact and simplicity, and the use cases of the method are clearly demonstrated.

  • Reviewer confidence

    Very confident (4)

  • [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.

    N/A

  • [Post rebuttal] Please justify your final decision from above.

    N/A



Author Feedback

N/A




Meta-Review

Meta-review #1

  • Your recommendation

    Provisional Accept

  • Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

    The three reviews show a clear consensus toward accept.



back to top