In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909114474430464 |
|---|---|
| author | Kuratov, Yuri Bulatov, Aydar Anokhin, Petr Sorokin, Dmitry Sorokin, Artyom Burtsev, Mikhail |
| author_facet | Kuratov, Yuri Bulatov, Aydar Anokhin, Petr Sorokin, Dmitry Sorokin, Artyom Burtsev, Mikhail |
| contents | This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10790 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss Kuratov, Yuri Bulatov, Aydar Anokhin, Petr Sorokin, Dmitry Sorokin, Artyom Burtsev, Mikhail Computation and Language Artificial Intelligence Machine Learning This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to $10^4$ elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to $11\times 10^6$ elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences. |
| title | In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.10790 |