Unsupervised Large Language Model Alignment for Information Retrieval via Contrastive Feedback
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913284245946368 |
|---|---|
| author | Dong, Qian Liu, Yiding Ai, Qingyao Wu, Zhijing Li, Haitao Liu, Yiqun Wang, Shuaiqiang Yin, Dawei Ma, Shaoping |
| author_facet | Dong, Qian Liu, Yiding Ai, Qingyao Wu, Zhijing Li, Haitao Liu, Yiqun Wang, Shuaiqiang Yin, Dawei Ma, Shaoping |
| contents | Large language models (LLMs) have demonstrated remarkable capabilities across various research domains, including the field of Information Retrieval (IR). However, the responses generated by off-the-shelf LLMs tend to be generic, i.e., cannot capture the distinctiveness of each document with similar content. This limits the performance of LLMs in IR because finding and distinguishing relevant documents from substantial similar documents is a typical problem in many IR tasks. To address this issue, we propose an unsupervised alignment method, namely Reinforcement Learning from Contrastive Feedback (RLCF), empowering LLMs to generate both high-quality and context-specific responses. Our approach constructs unsupervised contrastive feedback signals based on similar document groups, and adopts a reward function, named group-wise reciprocal rank, to optimize LLMs within a standard Proximal Policy Optimization. We conduct extensive experiments to evaluate the effectiveness of RLCF on LLMs built with different languages and parameter sizes on multiple downstream IR applications. RLCF significantly outperforms existing alignment methods, and RLCF-optimized LLMs demonstrate considerable improvement in generating responses with distinctiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_17078 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Unsupervised Large Language Model Alignment for Information Retrieval via Contrastive Feedback Dong, Qian Liu, Yiding Ai, Qingyao Wu, Zhijing Li, Haitao Liu, Yiqun Wang, Shuaiqiang Yin, Dawei Ma, Shaoping Information Retrieval Large language models (LLMs) have demonstrated remarkable capabilities across various research domains, including the field of Information Retrieval (IR). However, the responses generated by off-the-shelf LLMs tend to be generic, i.e., cannot capture the distinctiveness of each document with similar content. This limits the performance of LLMs in IR because finding and distinguishing relevant documents from substantial similar documents is a typical problem in many IR tasks. To address this issue, we propose an unsupervised alignment method, namely Reinforcement Learning from Contrastive Feedback (RLCF), empowering LLMs to generate both high-quality and context-specific responses. Our approach constructs unsupervised contrastive feedback signals based on similar document groups, and adopts a reward function, named group-wise reciprocal rank, to optimize LLMs within a standard Proximal Policy Optimization. We conduct extensive experiments to evaluate the effectiveness of RLCF on LLMs built with different languages and parameter sizes on multiple downstream IR applications. RLCF significantly outperforms existing alignment methods, and RLCF-optimized LLMs demonstrate considerable improvement in generating responses with distinctiveness. |
| title | Unsupervised Large Language Model Alignment for Information Retrieval via Contrastive Feedback |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2309.17078 |