An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915063067049984 |
|---|---|
| author | Janiak, Jett Rager, Can Dao, James Lau, Yeu-Tong |
| author_facet | Janiak, Jett Rager, Can Dao, James Lau, Yeu-Tong |
| contents | Prior work suggests that language models manage the limited bandwidth of the residual stream through a "memory management" mechanism, where certain attention heads and MLP layers clear residual stream directions set by earlier layers. Our study provides concrete evidence for this erasure phenomenon in a 4-layer transformer, identifying heads that consistently remove the output of earlier heads. We further demonstrate that direct logit attribution (DLA), a common technique for interpreting the output of intermediate transformer layers, can show misleading results by not accounting for erasure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07325 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L Janiak, Jett Rager, Can Dao, James Lau, Yeu-Tong Machine Learning Artificial Intelligence Prior work suggests that language models manage the limited bandwidth of the residual stream through a "memory management" mechanism, where certain attention heads and MLP layers clear residual stream directions set by earlier layers. Our study provides concrete evidence for this erasure phenomenon in a 4-layer transformer, identifying heads that consistently remove the output of earlier heads. We further demonstrate that direct logit attribution (DLA), a common technique for interpreting the output of intermediate transformer layers, can show misleading results by not accounting for erasure. |
| title | An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.07325 |