Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915418625540096 |
|---|---|
| author | Zheng, Carolina Beltran-Velez, Nicolas Karlekar, Sweta Shi, Claudia Nazaret, Achille Mallik, Asif Feder, Amir Blei, David M. |
| author_facet | Zheng, Carolina Beltran-Velez, Nicolas Karlekar, Sweta Shi, Claudia Nazaret, Achille Mallik, Asif Feder, Amir Blei, David M. |
| contents | Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic-based steering vectors. To properly evaluate MTM topics against word-list-based approaches, we propose \textit{topic judge}, an LLM-based pairwise comparison evaluation framework. Across five datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective steering of LLM outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_23220 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders Zheng, Carolina Beltran-Velez, Nicolas Karlekar, Sweta Shi, Claudia Nazaret, Achille Mallik, Asif Feder, Amir Blei, David M. Computation and Language Machine Learning Traditional topic models are effective at uncovering latent themes in large text collections. However, due to their reliance on bag-of-words representations, they struggle to capture semantically abstract features. While some neural variants use richer representations, they are similarly constrained by expressing topics as word lists, which limits their ability to articulate complex topics. We introduce Mechanistic Topic Models (MTMs), a class of topic models that operate on interpretable features learned by sparse autoencoders (SAEs). By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions. Moreover, uniquely among topic models, MTMs enable controllable text generation using topic-based steering vectors. To properly evaluate MTM topics against word-list-based approaches, we propose \textit{topic judge}, an LLM-based pairwise comparison evaluation framework. Across five datasets, MTMs match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective steering of LLM outputs. |
| title | Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.23220 |