Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.12331 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914837982871552 |
|---|---|
| author | Fei, Weizhi Niu, Xueyan Xie, Guoqing Zhang, Yanhua Bai, Bo Deng, Lei Han, Wei |
| author_facet | Fei, Weizhi Niu, Xueyan Xie, Guoqing Zhang, Yanhua Bai, Bo Deng, Lei Han, Wei |
| contents | Current Large Language Models (LLMs) face inherent limitations due to their pre-defined context lengths, which impede their capacity for multi-hop reasoning within extensive textual contexts. While existing techniques like Retrieval-Augmented Generation (RAG) have attempted to bridge this gap by sourcing external information, they fall short when direct answers are not readily available. We introduce a novel approach that re-imagines information retrieval through dynamic in-context editing, inspired by recent breakthroughs in knowledge editing. By treating lengthy contexts as malleable external knowledge, our method interactively gathers and integrates relevant information, thereby enabling LLMs to perform sophisticated reasoning steps. Experimental results demonstrate that our method effectively empowers context-limited LLMs, such as Llama2, to engage in multi-hop reasoning with improved performance, which outperforms state-of-the-art context window extrapolation methods and even compares favorably to more advanced commercial long-context models. Our interactive method not only enhances reasoning capabilities but also mitigates the associated training and computational costs, making it a pragmatic solution for enhancing LLMs' reasoning within expansive contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12331 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding Fei, Weizhi Niu, Xueyan Xie, Guoqing Zhang, Yanhua Bai, Bo Deng, Lei Han, Wei Computation and Language Artificial Intelligence Current Large Language Models (LLMs) face inherent limitations due to their pre-defined context lengths, which impede their capacity for multi-hop reasoning within extensive textual contexts. While existing techniques like Retrieval-Augmented Generation (RAG) have attempted to bridge this gap by sourcing external information, they fall short when direct answers are not readily available. We introduce a novel approach that re-imagines information retrieval through dynamic in-context editing, inspired by recent breakthroughs in knowledge editing. By treating lengthy contexts as malleable external knowledge, our method interactively gathers and integrates relevant information, thereby enabling LLMs to perform sophisticated reasoning steps. Experimental results demonstrate that our method effectively empowers context-limited LLMs, such as Llama2, to engage in multi-hop reasoning with improved performance, which outperforms state-of-the-art context window extrapolation methods and even compares favorably to more advanced commercial long-context models. Our interactive method not only enhances reasoning capabilities but also mitigates the associated training and computational costs, making it a pragmatic solution for enhancing LLMs' reasoning within expansive contexts. |
| title | Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.12331 |