OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909332348600320 |
|---|---|
| author | Zhang, Jintian Peng, Cheng Sun, Mengshu Chen, Xiang Liang, Lei Zhang, Zhiqiang Zhou, Jun Chen, Huajun Zhang, Ningyu |
| author_facet | Zhang, Jintian Peng, Cheng Sun, Mengshu Chen, Xiang Liang, Lei Zhang, Zhiqiang Zhou, Jun Chen, Huajun Zhang, Ningyu |
| contents | Despite the recent advancements in Large Language Models (LLMs), which have significantly enhanced the generative capabilities for various NLP tasks, LLMs still face limitations in directly handling retrieval tasks. However, many practical applications demand the seamless integration of both retrieval and generation. This paper introduces a novel and efficient One-pass Generation and retrieval framework (OneGen), designed to improve LLMs' performance on tasks that require both generation and retrieval. The proposed framework bridges the traditionally separate training approaches for generation and retrieval by incorporating retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass. We conduct experiments on two distinct types of composite tasks, RAG and Entity Linking, to validate the pluggability, effectiveness, and efficiency of OneGen in training and inference. Furthermore, our results show that integrating generation and retrieval within the same context preserves the generative capabilities of LLMs while improving retrieval performance. To the best of our knowledge, OneGen is the first to enable LLMs to conduct vector retrieval during the generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs Zhang, Jintian Peng, Cheng Sun, Mengshu Chen, Xiang Liang, Lei Zhang, Zhiqiang Zhou, Jun Chen, Huajun Zhang, Ningyu Computation and Language Artificial Intelligence Databases Information Retrieval Machine Learning Despite the recent advancements in Large Language Models (LLMs), which have significantly enhanced the generative capabilities for various NLP tasks, LLMs still face limitations in directly handling retrieval tasks. However, many practical applications demand the seamless integration of both retrieval and generation. This paper introduces a novel and efficient One-pass Generation and retrieval framework (OneGen), designed to improve LLMs' performance on tasks that require both generation and retrieval. The proposed framework bridges the traditionally separate training approaches for generation and retrieval by incorporating retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass. We conduct experiments on two distinct types of composite tasks, RAG and Entity Linking, to validate the pluggability, effectiveness, and efficiency of OneGen in training and inference. Furthermore, our results show that integrating generation and retrieval within the same context preserves the generative capabilities of LLMs while improving retrieval performance. To the best of our knowledge, OneGen is the first to enable LLMs to conduct vector retrieval during the generation. |
| title | OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs |
| topic | Computation and Language Artificial Intelligence Databases Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2409.05152 |