Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.17322 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913486844461056 |
|---|---|
| author | Pochinkov, Nicholas Pasero, Ben Shibayama, Skylar |
| author_facet | Pochinkov, Nicholas Pasero, Ben Shibayama, Skylar |
| contents | The use of transformer-based models is growing rapidly throughout society. With this growth, it is important to understand how they work, and in particular, how the attention mechanisms represent concepts. Though there are many interpretability methods, many look at models through their neuronal activations, which are poorly understood. We describe different lenses through which to view neuron activations, and investigate the effectiveness in language models and vision transformers through various methods of neural ablation: zero ablation, mean ablation, activation resampling, and a novel approach we term 'peak ablation'. Through experimental analysis, we find that in different regimes and models, each method can offer the lowest degradation of model performance compared to other methods, with resampling usually causing the most significant performance deterioration. We make our code available at https://github.com/nickypro/investigating-ablation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_17322 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering Pochinkov, Nicholas Pasero, Ben Shibayama, Skylar Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition 68T07 (Primary) 68T30, 68T50 (Secondary) I.2.4; I.2.6; I.2.7 The use of transformer-based models is growing rapidly throughout society. With this growth, it is important to understand how they work, and in particular, how the attention mechanisms represent concepts. Though there are many interpretability methods, many look at models through their neuronal activations, which are poorly understood. We describe different lenses through which to view neuron activations, and investigate the effectiveness in language models and vision transformers through various methods of neural ablation: zero ablation, mean ablation, activation resampling, and a novel approach we term 'peak ablation'. Through experimental analysis, we find that in different regimes and models, each method can offer the lowest degradation of model performance compared to other methods, with resampling usually causing the most significant performance deterioration. We make our code available at https://github.com/nickypro/investigating-ablation. |
| title | Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centering |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition 68T07 (Primary) 68T30, 68T50 (Secondary) I.2.4; I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2408.17322 |