DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909343995133952 |
|---|---|
| author | Zheng, Chuanyang Gao, Yihang Shi, Han Xiong, Jing Sun, Jiankai Li, Jingyao Huang, Minbin Ren, Xiaozhe Ng, Michael Jiang, Xin Li, Zhenguo Li, Yu |
| author_facet | Zheng, Chuanyang Gao, Yihang Shi, Han Xiong, Jing Sun, Jiankai Li, Jingyao Huang, Minbin Ren, Xiaozhe Ng, Michael Jiang, Xin Li, Zhenguo Li, Yu |
| contents | The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04798 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Zheng, Chuanyang Gao, Yihang Shi, Han Xiong, Jing Sun, Jiankai Li, Jingyao Huang, Minbin Ren, Xiaozhe Ng, Michael Jiang, Xin Li, Zhenguo Li, Yu Computation and Language The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance. |
| title | DAPE V2: Process Attention Score as Feature Map for Length Extrapolation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04798 |