Self-Retrieval: End-to-End Information Retrieval with One Large Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913569942011904 |
|---|---|
| author | Tang, Qiaoyu Chen, Jiawei Li, Zhuoqun Yu, Bowen Lu, Yaojie Fu, Cheng Yu, Haiyang Lin, Hongyu Huang, Fei He, Ben Han, Xianpei Sun, Le Li, Yongbin |
| author_facet | Tang, Qiaoyu Chen, Jiawei Li, Zhuoqun Yu, Bowen Lu, Yaojie Fu, Cheng Yu, Haiyang Lin, Hongyu Huang, Fei He, Ben Han, Xianpei Sun, Le Li, Yongbin |
| contents | The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00801 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Retrieval: End-to-End Information Retrieval with One Large Language Model Tang, Qiaoyu Chen, Jiawei Li, Zhuoqun Yu, Bowen Lu, Yaojie Fu, Cheng Yu, Haiyang Lin, Hongyu Huang, Fei He, Ben Han, Xianpei Sun, Le Li, Yongbin Information Retrieval Artificial Intelligence Computation and Language The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation. |
| title | Self-Retrieval: End-to-End Information Retrieval with One Large Language Model |
| topic | Information Retrieval Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2403.00801 |