Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917839687909376 |
|---|---|
| author | Dong, Zican Li, Junyi Men, Xin Zhao, Wayne Xin Wang, Bingbing Tian, Zhen Chen, Weipeng Wen, Ji-Rong |
| author_facet | Dong, Zican Li, Junyi Men, Xin Zhao, Wayne Xin Wang, Bingbing Tian, Zhen Chen, Weipeng Wen, Ji-Rong |
| contents | Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18009 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Context Window of Large Language Models via Decomposed Positional Vectors Dong, Zican Li, Junyi Men, Xin Zhao, Wayne Xin Wang, Bingbing Tian, Zhen Chen, Weipeng Wen, Ji-Rong Computation and Language Machine Learning Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length. |
| title | Exploring Context Window of Large Language Models via Decomposed Positional Vectors |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2405.18009 |