SEGMENT+: Long Text Processing with Short-Context Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929533624516608 |
|---|---|
| author | Shi, Wei Li, Shuang Yu, Kerun Chen, Jinglei Liang, Zujie Wu, Xinhui Qian, Yuxi Wei, Feng Zheng, Bo Liang, Jiaqing Chen, Jiangjie Xiao, Yanghua |
| author_facet | Shi, Wei Li, Shuang Yu, Kerun Chen, Jinglei Liang, Zujie Wu, Xinhui Qian, Yuxi Wei, Feng Zheng, Bo Liang, Jiaqing Chen, Jiangjie Xiao, Yanghua |
| contents | There is a growing interest in expanding the input capacity of language models (LMs) across various domains. However, simply increasing the context window does not guarantee robust performance across diverse long-input processing tasks, such as understanding extensive documents and extracting detailed information from lengthy and noisy data. In response, we introduce SEGMENT+, a general framework that enables LMs to handle extended inputs within limited context windows efficiently. SEGMENT+ utilizes structured notes and a filtering module to manage information flow, resulting in a system that is both controllable and interpretable. Our extensive experiments across various model sizes, focusing on long-document question-answering and Needle-in-a-Haystack tasks, demonstrate the effectiveness of SEGMENT+ in improving performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06519 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SEGMENT+: Long Text Processing with Short-Context Language Models Shi, Wei Li, Shuang Yu, Kerun Chen, Jinglei Liang, Zujie Wu, Xinhui Qian, Yuxi Wei, Feng Zheng, Bo Liang, Jiaqing Chen, Jiangjie Xiao, Yanghua Computation and Language There is a growing interest in expanding the input capacity of language models (LMs) across various domains. However, simply increasing the context window does not guarantee robust performance across diverse long-input processing tasks, such as understanding extensive documents and extracting detailed information from lengthy and noisy data. In response, we introduce SEGMENT+, a general framework that enables LMs to handle extended inputs within limited context windows efficiently. SEGMENT+ utilizes structured notes and a filtering module to manage information flow, resulting in a system that is both controllable and interpretable. Our extensive experiments across various model sizes, focusing on long-document question-answering and Needle-in-a-Haystack tasks, demonstrate the effectiveness of SEGMENT+ in improving performance. |
| title | SEGMENT+: Long Text Processing with Short-Context Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.06519 |