Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.01999 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913253173493760 |
|---|---|
| author | Sun, Si Zhang, Hanqing Liu, Zhiyuan Bao, Jie Song, Dawei |
| author_facet | Sun, Si Zhang, Hanqing Liu, Zhiyuan Bao, Jie Song, Dawei |
| contents | Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories. However, due to the paradigm discrepancy between text generation of LLM and DR, it is still an open challenge to integrate the retrieval and generation tasks in a shared LLM. In this paper, we propose an efficient LLM-Oriented Retrieval Tuner, namely LMORT, which decouples DR capacity from base LLM and non-invasively coordinates the optimally aligned and uniform layers of the LLM towards a unified DR space, achieving an efficient and effective DR without tuning the LLM itself. The extensive experiments on six BEIR datasets show that our approach could achieve competitive zero-shot retrieval performance compared to a range of strong DR models while maintaining the generation ability of LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01999 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-Oriented Retrieval Tuner Sun, Si Zhang, Hanqing Liu, Zhiyuan Bao, Jie Song, Dawei Computation and Language Dense Retrieval (DR) is now considered as a promising tool to enhance the memorization capacity of Large Language Models (LLM) such as GPT3 and GPT-4 by incorporating external memories. However, due to the paradigm discrepancy between text generation of LLM and DR, it is still an open challenge to integrate the retrieval and generation tasks in a shared LLM. In this paper, we propose an efficient LLM-Oriented Retrieval Tuner, namely LMORT, which decouples DR capacity from base LLM and non-invasively coordinates the optimally aligned and uniform layers of the LLM towards a unified DR space, achieving an efficient and effective DR without tuning the LLM itself. The extensive experiments on six BEIR datasets show that our approach could achieve competitive zero-shot retrieval performance compared to a range of strong DR models while maintaining the generation ability of LLM. |
| title | LLM-Oriented Retrieval Tuner |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.01999 |