Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.08570 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913666456092672 |
|---|---|
| author | Li, Kewei Kong, Yanwen Xu, Yiping Su, Jianlin Huang, Lan Zhang, Ruochi Zhou, Fengfeng |
| author_facet | Li, Kewei Kong, Yanwen Xu, Yiping Su, Jianlin Huang, Lan Zhang, Ruochi Zhou, Fengfeng |
| contents | Since the emergence of research on improving the length extrapolation capabilities of large language models in 2021, some studies have made modifications to the scaling factor in the scaled dot-product attention mechanism as part of their proposed methods without rigorous theoretical justifications. To fill this gap, we propose two new scaled temperatures based on information entropy invariance to enhance length extrapolation. First, a training-free method InfoScale is designed for dotproduct attention, and preserves focus on original tokens during length extrapolation by ensuring consistent entropy. Second, we theoretically analyze the impact of scaling (CosScale) on cosine attention. Experimental data demonstrates that combining InfoScale and CosScale achieves state-ofthe-art performance on the GAU-α model with a context window extended to 64 times the training length, and outperforms seven existing methods. Our analysis reveals that significantly increasing CosScale approximates the Windowed Attention, and highlights the significance of attention score dilution as a key challenge in long-range context handling. The code and data are available at https://github.com/HT-NEKO/ Information-Entropy-Invariance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08570 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms Li, Kewei Kong, Yanwen Xu, Yiping Su, Jianlin Huang, Lan Zhang, Ruochi Zhou, Fengfeng Computation and Language Since the emergence of research on improving the length extrapolation capabilities of large language models in 2021, some studies have made modifications to the scaling factor in the scaled dot-product attention mechanism as part of their proposed methods without rigorous theoretical justifications. To fill this gap, we propose two new scaled temperatures based on information entropy invariance to enhance length extrapolation. First, a training-free method InfoScale is designed for dotproduct attention, and preserves focus on original tokens during length extrapolation by ensuring consistent entropy. Second, we theoretically analyze the impact of scaling (CosScale) on cosine attention. Experimental data demonstrates that combining InfoScale and CosScale achieves state-ofthe-art performance on the GAU-α model with a context window extended to 64 times the training length, and outperforms seven existing methods. Our analysis reveals that significantly increasing CosScale approximates the Windowed Attention, and highlights the significance of attention score dilution as a key challenge in long-range context handling. The code and data are available at https://github.com/HT-NEKO/ Information-Entropy-Invariance. |
| title | Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.08570 |