Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.01601 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911942093832192 |
|---|---|
| author | Yan, Ruiqing Du, Xingbo Deng, Haoyu Zheng, Linghan Sun, Qiuzhuang Hu, Jifang Shao, Yuhang Jiang, Penghao Jiang, Jinrong Zhao, Lian |
| author_facet | Yan, Ruiqing Du, Xingbo Deng, Haoyu Zheng, Linghan Sun, Qiuzhuang Hu, Jifang Shao, Yuhang Jiang, Penghao Jiang, Jinrong Zhao, Lian |
| contents | With the advent of large models based on the Transformer architecture, researchers have observed an anomalous phenomenon in the Attention mechanism--there is a very high attention on the first element, which is prevalent across Transformer-based models. It is crucial to understand it for the development of techniques focusing on attention distribution, such as Key-Value (KV) Cache compression and infinite extrapolation; however, the latent cause leaves to be unknown. In this paper, we analyze such a phenomenon from the perspective of waiver phenomenon, which involves reducing the internal values of certain elements in the sequence, allowing them to absorb excess attention without affecting their contribution to information. In specific models, due to differences in positional encoding and attention patterns, we have found that the selection of waiver elements by the model can be categorized into two methods: positional-encoding-based and feature-distribution-within-elements-based. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01601 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unveiling and Controlling Anomalous Attention Distribution in Transformers Yan, Ruiqing Du, Xingbo Deng, Haoyu Zheng, Linghan Sun, Qiuzhuang Hu, Jifang Shao, Yuhang Jiang, Penghao Jiang, Jinrong Zhao, Lian Machine Learning Artificial Intelligence With the advent of large models based on the Transformer architecture, researchers have observed an anomalous phenomenon in the Attention mechanism--there is a very high attention on the first element, which is prevalent across Transformer-based models. It is crucial to understand it for the development of techniques focusing on attention distribution, such as Key-Value (KV) Cache compression and infinite extrapolation; however, the latent cause leaves to be unknown. In this paper, we analyze such a phenomenon from the perspective of waiver phenomenon, which involves reducing the internal values of certain elements in the sequence, allowing them to absorb excess attention without affecting their contribution to information. In specific models, due to differences in positional encoding and attention patterns, we have found that the selection of waiver elements by the model can be categorized into two methods: positional-encoding-based and feature-distribution-within-elements-based. |
| title | Unveiling and Controlling Anomalous Attention Distribution in Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.01601 |