List of Papers Browse by Subject Areas Author List
Abstract
Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D images from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R2 up to 0.941) and recover up to 87.8% of downstream performance using only 10 features (99.4% dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation, (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: https://github.com/pwesp/sail.
Links to Paper and Supplementary Materials
Main Paper (Open Access Version): https://papers.miccai.org/miccai-2026/paper/4497_paper.pdf
SharedIt Link: Not yet available
SpringerLink (DOI): Not yet available
Supplementary Material: Not Submitted
Link to the Code Repository
Link to the Dataset(s)
N/A
BibTex
@InProceedings{WesPhi_Sparse_MICCAI2026,
author = { Wesp, Philipp AND Holland, Robbie AND Sideri-Lampretsa, Vasiliki AND Gatidis, Sergios},
title = { { Sparse Autoencoders for Interpretable Medical Image Representation Learning } },
booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
year = {2026},
publisher = {Springer Nature Switzerland},
volume = {LNCS 16882},
month = {September},
page = {pending}
}
Reviews
Review #1
- Please describe the contribution of the paper
The paper studies whether sparse autoencoders (SAEs) can make the internal representations of medical imaging foundation models more interpretable without destroying their usefulness.
They apply SAEs to embeddings from two vision foundation models, one biomedical and one general-domain (DINOv3), using a large 2D dataset of CT and MRI slices. The idea is to replace dense, hard-to-interpret features with a sparse set of latent units that ideally correspond to cleaner concepts such as anatomy, modality, or scan orientation.
Empirically, the approach seem to preserve a large fraction of the original information while producing features that are more selective and easier to describe. The paper also tests these sparse features in a few interesting ways. reconstruction quality, recovery of downstream classification performance, retrieval of similar images, and automatic labeling of features with language models.
One of the more interesting results is that the general-domain model appears to yield more monosemantic sparse features than the biomedical model, which suggests that stronger interpretability does not necessarily require domain-specific pretraining.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Overall, I think the paper presents a really interesting approach to automatically discovering latent features that map onto meaningful semantic concepts. Although not an expert in the are of sparse representations, the approach seems novel The motivation is coherent, and the proposed use of sparse autoencoders as an interpretability layer is easy to understand from Fig. 1 and the introduction. Interesting experiments are conducted to show the strength of the approach. The paper includes reporting embedding reconstruction quality, downstream ROC-AUC recovery, sparse retrieval quality, and automatic concept labeling/judging. That is a real strength, because interpretability papers often lean too heavily on one proxy metric. These were performed on 3 models: biomedical FM (BiomedParse), a general-purpose FM (DINOv3), and a random baseline that is especially valuable.
The finding that DINOv3 outperforms BiomedParse on monosemanticity, despite lacking biomedical pretraining, is interesting
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Interpretability evidence is still mostly proxy-based (openly acknowledged in the conclusion: monosemanticity is based on metadata-derived organ labels, and feature descriptions are generated and judged by VLMs rather than humans) That makes the interpretability claim promising, but not fully validated. A clinician or radiologist evaluation would make the paper much stronger.
The paper sometimes comes close to implying that SAE features correspond to meaningful clinical concepts in a strong sense. The evidence supports that they often align with anatomy/modality/orientation, but not yet that they are consistently faithful, causal, or clinically actionable explanations. With CT and MRI being the applications, it would be interesting to see this translate to modern CT foundation models (CT-CLIP / MERLIN/ SPECTRE). Great follow-up research. Are there other baseline interpretability methods to compare against besides SAE?
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(5) Accept — should be accepted, independent of rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper’s presents a novel approach to automatically discover latent features that align with meaningful semantic. There is a clear motivation, strong experimental scope, and thoughtful multi-angle evaluation. Its biggest weaknesses are that the interpretability story is still validated mostly through proxies, and the clinically meaningful end use remains suggestive rather than fully demonstrated.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The other reviewers’ concerns seem to be adequately addressed.
Review #2
- Please describe the contribution of the paper
The paper proposes training sparse autoencoders (SAEs) on embeddings from foundation models to derive medically interpretable representations of CT and MRI images. The resulting sparse features are evaluated using fidelity metrics and a range of downstream tasks, including anatomical structure classification, image retrieval, and zero-shot language-driven image retrieval. Overall, the results suggest that sparse representations largely preserve the structure and performance of dense embeddings while improving interpretability, particularly for retrieval and language-driven tasks.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Comprehensive evaluation: The study extends prior work (Abdulaah et al.) by including MRI/CT data and evaluating multiple downstream tasks. Comparisons against dense embeddings provide a clear reference for fidelity and utility. The inclusion of language-driven image retrieval is particularly compelling, illustrating how interpretable embeddings can support semantically meaningful applications.
- Clear motivation and interpretability focus: The paper is clinically motivated and aims to align learned features with meaningful medical concepts.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
- Potential bias in automated feature interpretation: The use of the same VLM for both concept generation and concept judging raises concerns about confirmation bias. Since the model sees the same images in both stages, it is unclear to what extent the evaluation reflects genuine interpretability versus model self-consistency.
- Unclear candidate description selection: In the judging stage, four candidate textual descriptions are ranked per feature. However, the paper does not clearly specify how these candidates are selected. If candidates are sampled randomly, task difficulty may vary substantially across features; if candidates are selected via heuristics (e.g., semantic similarity), the procedure should be stated explicitly because it strongly affects evaluation difficulty and interpretability claims.
- Limited analysis on concepts generated by self-supervised models: While two self-supervised backbone models are compared on downstream tasks, there is no quantitative analysis of how similar (or different) the concepts generated by these models are. Such an analysis could clarify which semantic factors each representation captures and how these relate to downstream performance.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The submission has provided an anonymized link to the source code, dataset, or any other dependencies.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
- How are the four candidate descriptions selected for the automated judging stage (e.g., random sampling, semantic-similarity heuristics, or another strategy)?
- How are CT and MRI images preprocessed before embedding and SAE training, especially given that DINOv3 is pretrained on natural images while BiomedParse targets biomedical data? Differences in normalization, CT windowing, and MRI intensity scaling (non-quantitative) could affect the learned representations and the interpretability analysis.
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(4) Weak Accept — marginally above the acceptance threshold, but would not mind if rejected, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper focuses on the application aspect of a methodology proposed in previous work. However, the evaluation scheme is compelling, highlighting the use case of sparse embedding across various downstream tasks. Minor details can be clarified for reproducibility.
- Reviewer confidence
Confident but not absolutely certain (3)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Accept
- [Post rebuttal] Please justify your final decision from above.
The authors have addressed my concerns. This is an interesting and clinically relevant study that could be investigated and benchmarked more extensively in future work. I recommend accept.
Review #3
- Please describe the contribution of the paper
The main contribution of this paper is primarily empirical rather than methodological. The paper applies an existing sparse interpretability framework, a Matryoshka SAE, to medical vision embeddings, and studies whether sparse features learned from CT/MRI foundation-model representations can (i) reconstruct the original embeddings, (ii) retain downstream utility, (iii) support concept discovery and automatic language-based interpretation, and (iv) enable sparse-feature-based retrieval in a medical imaging setting. The value of the paper therefore lies in exploring the feasibility of bringing SAE-based representation analysis into the clinical imaging domain at relatively large scale, rather than in introducing a new technical method.
However, the paper does not appear to establish a sharply defined research gap or a domain-specific technical challenge that would require substantial adaptation beyond transferring an existing SAE pipeline to medical data. Core elements of the approach, sparse reconstruction, concept discovery from SAE features, and LLM/VLM-based concept interpretation, have already been demonstrated in prior non-medical or more general settings, and the paper does not clearly identify what is uniquely difficult about medical imaging in this context or what methodological modifications are necessary to address that difficulty. Thus, the contribution should be understood mainly as a domain application and empirical study of existing techniques in medical imaging, rather than as a methodologically novel advance.
- Please list the major strengths of the paper: you should highlight a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The interpretability evaluation is well designed and multi-dimensional: rather than relying on reconstruction or a single proxy score, it evaluates sparse representations through semantic retrieval fidelity, automated concept interpretation, and language-driven retrieval, which together provide a coherent and practically meaningful interpretability story.
2.The empirical study is extensive and systematic, including two distinct foundation-model embedding spaces, a random-weight baseline, and a broad sweep over dictionary sizes and sparsity patterns, with analysis across reconstruction, downstream utility, feature activity, monosemanticity, and top-N performance recovery.
3.The work yields several genuinely interesting findings: sparse codes preserve strong downstream performance with very few features, reconstruction fidelity is shown not to be sufficient for semantic utility, and DINOv3 unexpectedly exhibits stronger monosemanticity than the domain-specific BiomedParse model.
- Please list the major weaknesses of the paper. Please provide details: for instance, if you state that a formulation, way of using data, demonstration of clinical feasibility, or application is not novel, then you must provide specific references to prior work.
Major weaknesses: 1.The paper adopts a Matryoshka SAE, and the method description explicitly motivates a nested, coarse-to-fine hierarchy of representations. However, the experimental section does not provide a sufficient analysis of this hierarchical property. In particular, the paper does not address whether features from different levels differ in semantic granularity; whether earlier levels capture modality/orientation/coarse anatomical regions while later levels capture more specific organs or finer concepts; whether monosemanticity, retrieval quality, and language interpretability vary across levels; or how one should select levels for clinical queries, e.g., whether a coarse-to-fine strategy is preferable. These questions are important, especially since a hierarchical concept organization could be highly valuable in clinical settings.
2.The paper presents Automated Feature Interpretation as one of its interpretability evaluations, but this evaluation has notable limitations. First, it relies on an LLM/VLM-as-judge protocol. Although the judge model is separate from the generator, the evaluation still depends on model-internal prior knowledge rather than expert human assessment. Including sampled evaluation by clinical experts would substantially strengthen the paper’s contribution and credibility. Second, the setup uses one correct description and four distractors, but the paper does not specify how these distractors are selected. If the distractors are trivially different from the true concept, the task becomes much easier, which weakens the reliability of the reported results.
3.The paper also presents Language-Driven Image Retrieval as an interpretability demonstration, yet this section includes only a single case study rather than a quantitative evaluation over a broader query set. This weakens the evidence for clinical utility. More importantly, language-based image retrieval is not unique to SAE-based representations, but the paper does not compare against standard language-image retrieval baselines or metadata/text-supervised alternatives. Without such comparisons, it is difficult to assess the added value of the proposed pipeline beyond a proof-of-concept demonstration.
Points requiring clarification / writing issues: 1.In Section 3 (Experiments & Results) and the caption of Fig. 2, the paper refers to “96 configurations per FM.” However, the experimental setup describes only 4 dictionary-size families and 8 sparsity patterns, which suggests 32 configurations per FM rather than 96.The paper should clarify how the reported 96 configurations are counted.
2.The method section should provide more explicit formalization for quantities such as Monosemanticity scoring and Sparse Fingerprint Retrieval. For example, while M(f)=C(f)×S(f) is given, the exact formulations of the null-adjusted Jaccard coherence term and the normalized inverse-entropy specificity term are not fully specified.
3.In Fig. 2, the panel labeling/caption is inconsistent. The second row is mislabeled, and the caption currently refers to (E–G) even though there are four panels in the second row.
4.Fig. 3 is not explicitly referenced in the corresponding main-text discussion.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Please be aware that providing code and data is a plus, but not a requirement for acceptance.
The authors claimed to release the source code and/or dataset upon acceptance of the submission.
- Based on your review and your understanding of the MICCAI Scientific Code of Ethics, do you believe this submission may involve a potential ethics concern or violation?
N/A
- Optional: If you have any additional comments to share with the authors, please provide them here. Please also refer to our Reviewer’s guide on what makes a good review and pay specific attention to the different assessment criteria for the different paper categories: https://conferences.miccai.org/2026/en/REVIEWER-GUIDELINES.html
N/A
- Rate the paper on a scale of 1-6, 6 being the strongest (6-4: accept; 3-1: reject). Please use the entire range of the distribution. Spreading the score helps create a distribution for decision-making.
(3) Weak Reject — marginally below the acceptance threshold, but would not mind if accepted, dependent on rebuttal
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I do think the clinical direction is worthwhile. However, the strength of the claimed clinical contribution is limited by the current evaluation: the paper does not include expert human assessment, and some of the most clinically interesting claims, especially language-driven retrieval, are supported only by limited single-case demonstrations. As a result, while the paper provides useful exploratory evidence that SAE-style interpretability tools may transfer to medical imaging, it stops short of delivering strong clinical validation.
- Reviewer confidence
Very confident (4)
- [Post rebuttal] After reading the authors’ rebuttal, please state your final opinion of the paper.
Reject
- [Post rebuttal] Please justify your final decision from above.
The authors acknowledged in the rebuttal that the paper is framed as an application study. Therefore, the main contribution of this paper is primarily empirical rather than methodological. In this context, the authors should further strengthen the empirical contribution of the application. However, the two limitations of the paper remain clear.
1.For Automated Feature Interpretation, the authors clarify that distractors are uniformly sampled from real concepts, but this does not ensure they are hard negatives, so the choice of distractors may still affect the task difficulty. The evaluation also remains fully LLM/VLM-as-judge, without clinical expert validation (although acknowledged by the authors as future work), which weakens the evidence. The authors have partially addressed my concern, but it is not fully resolved.
2.Language-Driven Image Retrieval is supported by only a single case study and lacks quantitative evaluation over a broader query set. The authors state that this will be addressed in future work, but they present it as one of the main contributions in the abstract and conclusion. A single-case demonstration is therefore insufficient to support this claim. Although the authors promise to add comparisons with standard language-image retrieval baselines and argue that such comparisons are reasonable, they are not included in the current version.
The authors added some details on the Matryoshka hierarchy in the rebuttal, but the explanation remains limited. I believe such an analysis would strengthen the paper’s contribution. However, given the current content and space constraints, the limited treatment is understandable and should not be considered a major limitation.
Given the lack of methodological novelty and the two clear limitations in the empirical evaluation, I am overall inclined to reject the paper. However, considering its objective strengths, I view this as a borderline case, and I would not object if the paper is accepted.
Author Feedback
We thank all three reviewers for their constructive comments. We were glad the contribution resonated, and address the main points below:
1.Matryoshka hierarchy (R3) R3 asks whether the coarse-to-fine hierarchy is supported. In our trained SAEs, level-0 features fire on a median of 40-48k images and align with modality and scan plane, while level-3 features fire on only 150-1,200 images and align with specific organs, so semantic granularity increases with depth. Monosemanticity also rises with level: level-3 holds 2,610/3,682 (BP) and 2,361/3,577 (DINOv3) above-null coherent features. This supports a coarse-to-fine query strategy, shallow levels for modality and region, deep levels for organ-specific search. The per-level analysis was added to Sec. 3.2. 2.Automated feature interpretation (R2, R3) We apologize for the unclear distractor description (now in Sec. 2.2). Distractors are sampled uniformly from real feature concepts that are not-always-active with >10 activations. All five candidates are genuine VLM-generated descriptions, not synthetic or trivially different strings, so the judge must discriminate among real, often semantically close concepts. Chance is mean rank 3, so our 1.48 (BP) and 1.26 (DINOv3) are roughly half of chance. On VLM bias (R2): generation and judging are separate inference calls with different inputs (generator gets images and metadata, judge gets images and five texts). Because the distractors are also VLM-generated, any shared model prior aids all candidates equally and cannot favor the true one, so this is generic vision-language matching, not self-recognition. We agree clinician evaluation is the gold standard and plan it as the primary next step. The conclusion already frames our scoring as “scalable but proxy-based evidence” complementing expert panels.
3.Language-driven image retrieval (R3) R3 notes one query and no text-supervised baseline (BiomedCLIP/PubMedCLIP). We agree such a zero-shot baseline is a valid comparison and will add it. The contribution is orthogonal: text-supervised retrieval maps text to images but yields no inspectable features, whereas our pipeline routes retrieval through named, auditable sparse concepts. Broader-query evaluation is already flagged as future work, and we now also add a planned clinician-validated query benchmark for whole-body CT and MRI.
4.Cross-FM concept comparison (R2) Concepts share one anatomical vocabulary, but the top-activating images they are described from are model-specific, so concept quality tracks sparse-feature quality: DINOv3 is more monosemantic (0.356-0.714 vs BP 0.036-0.394, Sec. 3.2), with the same gap in Top-N recovery and language retrieval. A quantitative cross-FM concept-similarity study is planned.
5.Methodological clarifications (R2, R3) “96 configurations” is 32 per FM × 3 models, corrected throughout. Full formulas for C(f), S(f), M=C(f)×S(f) and the null baseline were added to Sec. 2.A preprocessing paragraph (R2) was added to Sec. 3.1: CT uses 38 study-type-dependent HU windows rescaled to [0,255]. MRI uses in-plane isotropic resampling and 1st-99th percentile clipping. Slices are center-cropped, resized to 1024×1024 (Lanczos), 3-channel replicated, ImageNet-normalized. The Fig. 2 and Fig. 3 issues were corrected.
6.Novelty (R3) R3 asks what is domain-specific here. Unlike chest radiographs with paired reports [1], whole-body CT/MRI has no paired text, so prior medical-SAE label supervision does not transfer. We therefore contribute: (a) VLM- and segmentation-based bootstrapping of concept labels with a null-adjusted monosemanticity metric that makes interpretability measurable without text supervision. (b) A three-way FM comparison including a random-weight baseline that isolates learned structure from architecture. (c) A configuration-ranking framework exposing the monosemanticity-performance trade-off. We accept the application-study framing and now state these elements explicitly in the introduction.
Meta-Review
Meta-review #1
- Your recommendation
Invite for Rebuttal
- Please justify your decision. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper presents a promising application of sparse autoencoders for interpretable medical image representation learning, with a clear motivation and broad evaluation across reconstruction, downstream utility, sparse retrieval, concept interpretation, and language-driven retrieval. Reviewers appreciated the multi-model comparison, random baseline, and the finding that DINOv3 yields more monosemantic features than the biomedical model. However, the rebuttal should clearly address several concerns. For example, the novelty beyond applying existing SAE techniques to medical imaging; how the claimed Matryoshka coarse-to-fine hierarchy is supported by the current results; the automated interpretation protocol, including distractor selection and possible VLM/LLM evaluation bias; the added value of language-driven retrieval given the lack of comparison to standard retrieval or metadata/text-supervised alternatives; and details on preprocessing, monosemanticity scoring, and the reported number of configurations.
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
Although the work has limited methodological novelty and the interpretability evidence remains partly proxy-based, the paper presents a timely and well-executed application of sparse autoencoders to medical image foundation-model representations. The rebuttal sufficiently clarifies several concerns, and the empirical findings are interesting enough to merit discussion at the conference.
Meta-review #2
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
The paper proposes to apply an existing method to explore the clinical relevance of features learned from CT/MRI foundation-model representations. In particular, the authors train sparse autoencoders on embeddings to (i) reconstruct the original embeddings, (ii) retain downstream utility, (iii) support concept discovery and automatic language-based interpretation, and (iv) enable sparse-feature-based retrieval in a medical imaging setting. Overall, the paper is well written and the motivation is clear. The reviewers raised a few concerns in the first round of revisions, particularly regarding the use of VLM-based evaluation rather than human-level evaluation and the analysis of features extracted from hierarchical embeddings. Other minor points were raised. During the rebuttal, the authors answered these concerns and justified their use of VLM-based evaluation. In addition, they provide more details on the per-level analysis of the hierarchical embeddings. They also clarify several minor points that were raised by R2 and R3. Overall, the reviewers seemed quite satisfied by the rebuttal, but R3 remained skeptical and felt that its concerns were not fully addressed. However, given the interdiction of adding new results to the paper after rebuttal, I believe the authors did their best to clarify their choices and provide evidence for their claims. Therefore, I recommend acceptance.
Meta-review #3
- After you have reviewed the rebuttal and updated reviews, please provide your recommendation based on all reviews and the authors’ rebuttal.
Accept
- Please justify your recommendation.
This paper received two rather clear Accept recommendations post-rebuttal and one Reject recommendation, which however, acknowledges the strengths of the paper.
Overall this is an interesting work for the MICCAI community towards interpretability of foundation model embeddings with a solid evaluation that can be built on in future studies.