Supervised sparse auto-encoders for interpretable and compositional representations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914573571850240 |
|---|---|
| author | Harzli, Ouns El Wallner, Hugo Nam, Yoonsoo Tao, Haixuan Xavier |
| author_facet | Harzli, Ouns El Wallner, Hugo Nam, Yoonsoo Tao, Haixuan Xavier |
| contents | Sparse auto-encoders (SAEs) have re-emerged as a prominent method for mechanistic interpretability, yet they face two significant challenges: the non-smoothness of the $L_1$ penalty, which hinders reconstruction and scalability, and a lack of alignment between learned features and human semantics. In this paper, we address these limitations by adapting unconstrained feature models, a mathematical framework from neural collapse theory, and by supervising the task. We supervise (decoder-only) SAEs to reconstruct feature vectors by jointly learning sparse concept embeddings and decoder weights. Validated on Stable Diffusion 3.5, our approach demonstrates compositional generalization, successfully reconstructing images with concept combinations unseen during training, and enabling feature-level intervention for semantic image editing without prompt modification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00924 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Supervised sparse auto-encoders for interpretable and compositional representations Harzli, Ouns El Wallner, Hugo Nam, Yoonsoo Tao, Haixuan Xavier Artificial Intelligence Sparse auto-encoders (SAEs) have re-emerged as a prominent method for mechanistic interpretability, yet they face two significant challenges: the non-smoothness of the $L_1$ penalty, which hinders reconstruction and scalability, and a lack of alignment between learned features and human semantics. In this paper, we address these limitations by adapting unconstrained feature models, a mathematical framework from neural collapse theory, and by supervising the task. We supervise (decoder-only) SAEs to reconstruct feature vectors by jointly learning sparse concept embeddings and decoder weights. Validated on Stable Diffusion 3.5, our approach demonstrates compositional generalization, successfully reconstructing images with concept combinations unseen during training, and enabling feature-level intervention for semantic image editing without prompt modification. |
| title | Supervised sparse auto-encoders for interpretable and compositional representations |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.00924 |