Are Long-LLMs A Necessity For Long-Context Tasks?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917674167042048 |
|---|---|
| author | Qian, Hongjin Liu, Zheng Zhang, Peitian Mao, Kelong Zhou, Yujia Chen, Xu Dou, Zhicheng |
| author_facet | Qian, Hongjin Liu, Zheng Zhang, Peitian Mao, Kelong Zhou, Yujia Chen, Xu Dou, Zhicheng |
| contents | The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-context tasks, as common long-context tasks are short-context solvable, i.e. they can be solved by purely working with oracle short-contexts within the long-context tasks' inputs. On top of this argument, we propose a framework called LC-Boost (Long-Context Bootstrapper), which enables a short-LLM to address the long-context tasks in a bootstrapping manner. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, LC-Boost can serve as a general framework to handle diversified long-context processing problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where LC-Boost is able to achieve a substantially improved performance with a much smaller consumption of resource. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15318 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are Long-LLMs A Necessity For Long-Context Tasks? Qian, Hongjin Liu, Zheng Zhang, Peitian Mao, Kelong Zhou, Yujia Chen, Xu Dou, Zhicheng Computation and Language Artificial Intelligence The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-context tasks, as common long-context tasks are short-context solvable, i.e. they can be solved by purely working with oracle short-contexts within the long-context tasks' inputs. On top of this argument, we propose a framework called LC-Boost (Long-Context Bootstrapper), which enables a short-LLM to address the long-context tasks in a bootstrapping manner. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, LC-Boost can serve as a general framework to handle diversified long-context processing problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where LC-Boost is able to achieve a substantially improved performance with a much smaller consumption of resource. |
| title | Are Long-LLMs A Necessity For Long-Context Tasks? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.15318 |