Do Sparse Autoencoders Capture Concept Manifolds?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914521696698368 |
|---|---|
| author | Bhalla, Usha Fel, Thomas Rager, Can Feucht, Sheridan Haklay, Tal Wurgaft, Daniel Boppana, Siddharth Kowal, Matthew Shyam, Vasudev Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh |
| author_facet | Bhalla, Usha Fel, Thomas Rager, Can Feucht, Sheridan Haklay, Tal Wurgaft, Daniel Boppana, Siddharth Kowal, Matthew Shyam, Vasudev Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh |
| contents | Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_28119 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do Sparse Autoencoders Capture Concept Manifolds? Bhalla, Usha Fel, Thomas Rager, Can Feucht, Sheridan Haklay, Tal Wurgaft, Daniel Boppana, Siddharth Kowal, Matthew Shyam, Vasudev Merullo, Jack Geiger, Atticus Lubana, Ekdeep Singh Machine Learning Artificial Intelligence Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability. |
| title | Do Sparse Autoencoders Capture Concept Manifolds? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.28119 |