Positional Encoding via Token-Aware Phase Attention
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911668235141120 |
|---|---|
| author | Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong |
| author_facet | Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong |
| contents | We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12635 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Positional Encoding via Token-Aware Phase Attention Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong Computation and Language Artificial Intelligence We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines. |
| title | Positional Encoding via Token-Aware Phase Attention |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12635 |