Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910907229011968 |
|---|---|
| author | Ji, Luo Guo, Feixiang Chen, Teng Gu, Qingqing Wang, Xiaoyu Xi, Ningyuan Wang, Yihong Yu, Peng Zhao, Yue Lei, Hongyang Jiang, Zhonglin Chen, Yong |
| author_facet | Ji, Luo Guo, Feixiang Chen, Teng Gu, Qingqing Wang, Xiaoyu Xi, Ningyuan Wang, Yihong Yu, Peng Zhao, Yue Lei, Hongyang Jiang, Zhonglin Chen, Yong |
| contents | Despite the recent advancement in Retrieval-Augmented Generation (RAG) systems, most retrieval methodologies are often developed for factual retrieval, which assumes query and positive documents are semantically similar. In this paper, we instead propose and study a more challenging type of retrieval task, called hidden rationale retrieval, in which query and document are not similar but can be inferred by reasoning chains, logic relationships, or empirical experiences. To address such problems, an instruction-tuned Large language model (LLM) with a cross-encoder architecture could be a reasonable choice. To further strengthen pioneering LLM-based retrievers, we design a special instruction that transforms the retrieval task into a generative task by prompting LLM to answer a binary-choice question. The model can be fine-tuned with direct preference optimization (DPO). The framework is also optimized for computational efficiency with no performance degradation. We name this retrieval framework by RaHoRe and verify its zero-shot and fine-tuned performance superiority on Emotional Support Conversation (ESC), compared with previous retrieval works. Our study suggests the potential to employ LLM as a foundation for a wider scope of retrieval tasks. Our codes, models, and datasets are available on https://github.com/flyfree5/LaHoRe. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16615 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval Ji, Luo Guo, Feixiang Chen, Teng Gu, Qingqing Wang, Xiaoyu Xi, Ningyuan Wang, Yihong Yu, Peng Zhao, Yue Lei, Hongyang Jiang, Zhonglin Chen, Yong Information Retrieval Computation and Language Machine Learning Despite the recent advancement in Retrieval-Augmented Generation (RAG) systems, most retrieval methodologies are often developed for factual retrieval, which assumes query and positive documents are semantically similar. In this paper, we instead propose and study a more challenging type of retrieval task, called hidden rationale retrieval, in which query and document are not similar but can be inferred by reasoning chains, logic relationships, or empirical experiences. To address such problems, an instruction-tuned Large language model (LLM) with a cross-encoder architecture could be a reasonable choice. To further strengthen pioneering LLM-based retrievers, we design a special instruction that transforms the retrieval task into a generative task by prompting LLM to answer a binary-choice question. The model can be fine-tuned with direct preference optimization (DPO). The framework is also optimized for computational efficiency with no performance degradation. We name this retrieval framework by RaHoRe and verify its zero-shot and fine-tuned performance superiority on Emotional Support Conversation (ESC), compared with previous retrieval works. Our study suggests the potential to employ LLM as a foundation for a wider scope of retrieval tasks. Our codes, models, and datasets are available on https://github.com/flyfree5/LaHoRe. |
| title | Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval |
| topic | Information Retrieval Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2412.16615 |