Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.12635 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911668235141120 |
|---|---|
| author | Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong |
| author_facet | Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong |
| contents | We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12635 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Positional Encoding via Token-Aware Phase Attention Wang, Yu Shen, Sheng Munos, Rémi Zhan, Hongyuan Tian, Yuandong Computation and Language Artificial Intelligence We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scores that limits RoPE's ability to model long-context. RoPE extension methods may alleviate this issue, but they typically require post-hoc adjustments after pretraining, such as rescaling or hyperparameters retuning. This paper introduces Token-Aware Phase Attention (TAPA), a new positional encoding method that incorporates a learnable phase function into the attention mechanism. TAPA preserves token interactions over long range, extends to longer contexts with direct and light continual pretraining, extrapolates to unseen lengths, and attains substantially lower perplexity and stronger retrieval performance in the long-context regime than RoPE-style baselines. |
| title | Positional Encoding via Token-Aware Phase Attention |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12635 |