Functional Interpolation for Relative Positions Improves Long Context Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916144338698240 |
|---|---|
| author | Li, Shanda You, Chong Guruganesh, Guru Ainslie, Joshua Ontanon, Santiago Zaheer, Manzil Sanghai, Sumit Yang, Yiming Kumar, Sanjiv Bhojanapalli, Srinadh |
| author_facet | Li, Shanda You, Chong Guruganesh, Guru Ainslie, Joshua Ontanon, Santiago Zaheer, Manzil Sanghai, Sumit Yang, Yiming Kumar, Sanjiv Bhojanapalli, Srinadh |
| contents | Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_04418 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Functional Interpolation for Relative Positions Improves Long Context Transformers Li, Shanda You, Chong Guruganesh, Guru Ainslie, Joshua Ontanon, Santiago Zaheer, Manzil Sanghai, Sumit Yang, Yiming Kumar, Sanjiv Bhojanapalli, Srinadh Machine Learning Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks. |
| title | Functional Interpolation for Relative Positions Improves Long Context Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.04418 |