Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911779710304256 |
|---|---|
| author | Qiu, Zihan Huang, Zeyu Huang, Youcheng Fu, Jie |
| author_facet | Qiu, Zihan Huang, Zeyu Huang, Youcheng Fu, Jie |
| contents | The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12233 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers Qiu, Zihan Huang, Zeyu Huang, Youcheng Fu, Jie Computation and Language The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$. |
| title | Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.12233 |