Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.22447 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912861836541952 |
|---|---|
| author | Liu, Yiting Deng, Zhi-Hong |
| author_facet | Liu, Yiting Deng, Zhi-Hong |
| contents | Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation methods infer feature semantics from activation patterns, but overlook that features are trained to reconstruct activations that serve computational roles in the forward pass. We introduce a novel weight-based interpretation framework that measures functional effects through direct weight interactions, requiring no activation data. Through three experiments on Gemma-2 and Llama-3.1 models, we demonstrate that (1) 1/4 of features directly predict output tokens, (2) features actively participate in attention mechanisms with depth-dependent structure, and (3) semantic and non-semantic feature populations exhibit distinct distribution profiles in attention circuits. Our analysis provides the missing out-of-context half of SAE feature interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22447 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features Liu, Yiting Deng, Zhi-Hong Machine Learning Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation methods infer feature semantics from activation patterns, but overlook that features are trained to reconstruct activations that serve computational roles in the forward pass. We introduce a novel weight-based interpretation framework that measures functional effects through direct weight interactions, requiring no activation data. Through three experiments on Gemma-2 and Llama-3.1 models, we demonstrate that (1) 1/4 of features directly predict output tokens, (2) features actively participate in attention mechanisms with depth-dependent structure, and (3) semantic and non-semantic feature populations exhibit distinct distribution profiles in attention circuits. Our analysis provides the missing out-of-context half of SAE feature interpretability. |
| title | Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.22447 |