Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914836146814976 |
|---|---|
| author | He, Zhenyu Feng, Guhao Luo, Shengjie Yang, Kai Wang, Liwei Xu, Jingjing Zhang, Zhi Yang, Hongxia He, Di |
| author_facet | He, Zhenyu Feng, Guhao Luo, Shengjie Yang, Kai Wang, Liwei Xu, Jingjing Zhang, Zhi Yang, Hongxia He, Di |
| contents | In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16421 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation He, Zhenyu Feng, Guhao Luo, Shengjie Yang, Kai Wang, Liwei Xu, Jingjing Zhang, Zhi Yang, Hongxia He, Di Machine Learning Artificial Intelligence Computation and Language In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities. |
| title | Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2401.16421 |