Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.20938 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909597103554560 |
|---|---|
| author | He, Zhengfu Wang, Junxuan Lin, Rui Ge, Xuyang Shu, Wentao Tang, Qiong Zhang, Junping Qiu, Xipeng |
| author_facet | He, Zhengfu Wang, Junxuan Lin, Rui Ge, Xuyang Shu, Wentao Tang, Qiong Zhang, Junping Qiu, Xipeng |
| contents | We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20938 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition He, Zhengfu Wang, Junxuan Lin, Rui Ge, Xuyang Shu, Wentao Tang, Qiong Zhang, Junping Qiu, Xipeng Machine Learning Computation and Language We propose Low-Rank Sparse Attention (Lorsa), a sparse replacement model of Transformer attention layers to disentangle original Multi Head Self Attention (MHSA) into individually comprehensible components. Lorsa is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions. We show that Lorsa heads find cleaner and finer-grained versions of previously discovered MHSA behaviors like induction heads, successor heads and attention sink behavior (i.e., heavily attending to the first token). Lorsa and Sparse Autoencoder (SAE) are both sparse dictionary learning methods applied to different Transformer components, and lead to consistent findings in many ways. For instance, we discover a comprehensive family of arithmetic-specific Lorsa heads, each corresponding to an atomic operation in Llama-3.1-8B. Automated interpretability analysis indicates that Lorsa achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties, especially for features computed collectively by multiple MHSA heads. We also conduct extensive experiments on architectural design ablation, Lorsa scaling law and error analysis. |
| title | Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2504.20938 |