On the token distance modeling ability of higher RoPE attention dimension
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914980158242816 |
|---|---|
| author | Hong, Xiangyu Jiang, Che Qi, Biqing Meng, Fandong Yu, Mo Zhou, Bowen Zhou, Jie |
| author_facet | Hong, Xiangyu Jiang, Che Qi, Biqing Meng, Fandong Yu, Mo Zhou, Bowen Zhou, Jie |
| contents | Length extrapolation algorithms based on Rotary position embedding (RoPE) have shown promising results in extending the context length of language models. However, understanding how position embedding can capture longer-range contextual information remains elusive. Based on the intuition that different dimensions correspond to different frequency of changes in RoPE encoding, we conducted a dimension-level analysis to investigate the correlation between a hidden dimension of an attention head and its contribution to capturing long-distance dependencies. Using our correlation metric, we identified a particular type of attention heads, which we named Positional Heads, from various length-extrapolated models. These heads exhibit a strong focus on long-range information interaction and play a pivotal role in long input processing, as evidence by our ablation. We further demonstrate the correlation between the efficiency of length extrapolation and the extension of the high-dimensional attention allocation of these heads. The identification of Positional Heads provides insights for future research in long-text comprehension. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08703 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the token distance modeling ability of higher RoPE attention dimension Hong, Xiangyu Jiang, Che Qi, Biqing Meng, Fandong Yu, Mo Zhou, Bowen Zhou, Jie Computation and Language Artificial Intelligence Length extrapolation algorithms based on Rotary position embedding (RoPE) have shown promising results in extending the context length of language models. However, understanding how position embedding can capture longer-range contextual information remains elusive. Based on the intuition that different dimensions correspond to different frequency of changes in RoPE encoding, we conducted a dimension-level analysis to investigate the correlation between a hidden dimension of an attention head and its contribution to capturing long-distance dependencies. Using our correlation metric, we identified a particular type of attention heads, which we named Positional Heads, from various length-extrapolated models. These heads exhibit a strong focus on long-range information interaction and play a pivotal role in long input processing, as evidence by our ablation. We further demonstrate the correlation between the efficiency of length extrapolation and the extension of the high-dimensional attention allocation of these heads. The identification of Positional Heads provides insights for future research in long-text comprehension. |
| title | On the token distance modeling ability of higher RoPE attention dimension |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.08703 |