Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.15019 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912122125942784 |
|---|---|
| author | Giglemiani, Giorgi Petrova, Nora Mangat, Chatrik Singh Janiak, Jett Heimersheim, Stefan |
| author_facet | Giglemiani, Giorgi Petrova, Nora Mangat, Chatrik Singh Janiak, Jett Heimersheim, Stefan |
| contents | Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15019 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Synthetic Activations composed of SAE Latents in GPT-2 Giglemiani, Giorgi Petrova, Nora Mangat, Chatrik Singh Janiak, Jett Heimersheim, Stefan Machine Learning Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations. |
| title | Evaluating Synthetic Activations composed of SAE Latents in GPT-2 |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2409.15019 |