Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2406.13632 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912503248715776 |
|---|---|
| author | Cattan, Arie Jacovi, Alon Fabrikant, Alex Herzig, Jonathan Aharoni, Roee Rashkin, Hannah Marcus, Dror Hassidim, Avinatan Matias, Yossi Szpektor, Idan Caciularu, Avi |
| author_facet | Cattan, Arie Jacovi, Alon Fabrikant, Alex Herzig, Jonathan Aharoni, Roee Rashkin, Hannah Marcus, Dror Hassidim, Avinatan Matias, Yossi Szpektor, Idan Caciularu, Avi |
| contents | Despite recent advancements in Large Language Models (LLMs), their performance on tasks involving long contexts remains sub-optimal. In this work, we propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks by recycling contexts. Specifically, given a long input context (1-3k tokens) and a query, we generate additional query-output pairs from the given context as few-shot examples, while introducing the context only once. This ensures that the demonstrations are leveraging the same context as the target query while only adding a small number of tokens to the prompt. We further enhance each demonstration by instructing the model to explicitly identify the relevant paragraphs before the answer, which improves performance while providing fine-grained attribution to the answer source. We apply our method on multiple LLMs and obtain substantial improvements (+16 absolute points on average across models) on various QA datasets with long context. Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA using our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13632 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DoubleDipper: Improving Long-Context LLMs via Context Recycling Cattan, Arie Jacovi, Alon Fabrikant, Alex Herzig, Jonathan Aharoni, Roee Rashkin, Hannah Marcus, Dror Hassidim, Avinatan Matias, Yossi Szpektor, Idan Caciularu, Avi Computation and Language Despite recent advancements in Large Language Models (LLMs), their performance on tasks involving long contexts remains sub-optimal. In this work, we propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks by recycling contexts. Specifically, given a long input context (1-3k tokens) and a query, we generate additional query-output pairs from the given context as few-shot examples, while introducing the context only once. This ensures that the demonstrations are leveraging the same context as the target query while only adding a small number of tokens to the prompt. We further enhance each demonstration by instructing the model to explicitly identify the relevant paragraphs before the answer, which improves performance while providing fine-grained attribution to the answer source. We apply our method on multiple LLMs and obtain substantial improvements (+16 absolute points on average across models) on various QA datasets with long context. Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA using our approach. |
| title | DoubleDipper: Improving Long-Context LLMs via Context Recycling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.13632 |