Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912059117010944 |
|---|---|
| author | Zhao, Liang Feng, Xiachong Feng, Xiaocheng Zhong, Weihong Xu, Dongliang Yang, Qing Liu, Hongtao Qin, Bing Liu, Ting |
| author_facet | Zhao, Liang Feng, Xiachong Feng, Xiaocheng Zhong, Weihong Xu, Dongliang Yang, Qing Liu, Hongtao Qin, Bing Liu, Ting |
| contents | Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they cannot perform length extrapolation to handle long sequences, which severely hinders their application in scenarios demanding long input sequences such as legal or scientific documents. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, we aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_17044 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding Zhao, Liang Feng, Xiachong Feng, Xiaocheng Zhong, Weihong Xu, Dongliang Yang, Qing Liu, Hongtao Qin, Bing Liu, Ting Computation and Language Built upon the Transformer, large language models (LLMs) have captured worldwide attention due to their remarkable abilities. Nevertheless, all Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, namely, they cannot perform length extrapolation to handle long sequences, which severely hinders their application in scenarios demanding long input sequences such as legal or scientific documents. Thus, numerous methods have emerged to enhance the length extrapolation of Transformers. Despite the great research efforts, a systematic survey is still lacking. To fill this gap, we delve into these advances in a unified notation from the perspective of positional encoding (PE), as it has been considered the primary factor on length extrapolation. Specifically, we begin with extrapolatable PEs that have dominated this research field. Then, we dive into extrapolation methods based on them, covering position interpolation and randomized position methods. Finally, several challenges and future directions in this area are highlighted. Through this survey, we aim to enable the reader to gain a deep understanding of existing methods and provide stimuli for future research. |
| title | Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2312.17044 |