Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909687567351808 |
|---|---|
| author | Hua, Ermo Jiang, Che Lv, Xingtai Zhang, Kaiyan Sun, Youbang Fan, Yuchen Zhu, Xuekai Qi, Biqing Ding, Ning Zhou, Bowen |
| author_facet | Hua, Ermo Jiang, Che Lv, Xingtai Zhang, Kaiyan Sun, Youbang Fan, Yuchen Zhu, Xuekai Qi, Biqing Ding, Ning Zhou, Bowen |
| contents | Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components brought by time-domain truncation. Building on our observations, we propose Fourier Position Embedding (FoPE), which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization. FoPE constructs \textit{Fourier Series} and zero-outs the destructive frequency components, increasing model robustness against the spectrum damage. Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains a more stable performance compared to other baselines. Several analyses and ablations bring further support to our method and theoretical modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17739 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization Hua, Ermo Jiang, Che Lv, Xingtai Zhang, Kaiyan Sun, Youbang Fan, Yuchen Zhu, Xuekai Qi, Biqing Ding, Ning Zhou, Bowen Artificial Intelligence Computation and Language Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components brought by time-domain truncation. Building on our observations, we propose Fourier Position Embedding (FoPE), which enhances attention's frequency-domain properties to improve both its periodic extension and length generalization. FoPE constructs \textit{Fourier Series} and zero-outs the destructive frequency components, increasing model robustness against the spectrum damage. Experiments across various model scales and benchmarks show that, within varying context windows, FoPE maintains a more stable performance compared to other baselines. Several analyses and ablations bring further support to our method and theoretical modeling. |
| title | Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.17739 |