Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911330630369280 |
|---|---|
| author | Kurochkin, Vadim Aksenov, Yaroslav Laptev, Daniil Gavrilov, Daniil Balagansky, Nikita |
| author_facet | Kurochkin, Vadim Aksenov, Yaroslav Laptev, Daniil Gavrilov, Daniil Balagansky, Nikita |
| contents | Sparse Autoencoders (SAEs) have demonstrated significant promise in interpreting the hidden states of language models by decomposing them into interpretable latent directions. However, training and interpreting SAEs at scale remains challenging, especially when large dictionary sizes are used. While decoders can leverage sparse-aware kernels for efficiency, encoders still require computationally intensive linear operations with large output dimensions. To address this, we propose KronSAE, a novel architecture that factorizes the latent representation via Kronecker product decomposition, drastically reducing memory and computational overhead. Furthermore, we introduce mAND, a differentiable activation function approximating the binary AND operation, which improves interpretability and performance in our factorized framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22255 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders Kurochkin, Vadim Aksenov, Yaroslav Laptev, Daniil Gavrilov, Daniil Balagansky, Nikita Machine Learning Computation and Language Sparse Autoencoders (SAEs) have demonstrated significant promise in interpreting the hidden states of language models by decomposing them into interpretable latent directions. However, training and interpreting SAEs at scale remains challenging, especially when large dictionary sizes are used. While decoders can leverage sparse-aware kernels for efficiency, encoders still require computationally intensive linear operations with large output dimensions. To address this, we propose KronSAE, a novel architecture that factorizes the latent representation via Kronecker product decomposition, drastically reducing memory and computational overhead. Furthermore, we introduce mAND, a differentiable activation function approximating the binary AND operation, which improves interpretability and performance in our factorized framework. |
| title | Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2505.22255 |