Context-DPO: Aligning Language Models for Context-Faithfulness
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913620463452160 |
|---|---|
| author | Bi, Baolong Huang, Shaohan Wang, Yiwei Yang, Tianchi Zhang, Zihan Huang, Haizhen Mei, Lingrui Fang, Junfeng Li, Zehao Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Liu, Shenghua |
| author_facet | Bi, Baolong Huang, Shaohan Wang, Yiwei Yang, Tianchi Zhang, Zihan Huang, Haizhen Mei, Lingrui Fang, Junfeng Li, Zehao Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Liu, Shenghua |
| contents | Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at https://github.com/byronBBL/Context-DPO |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15280 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Context-DPO: Aligning Language Models for Context-Faithfulness Bi, Baolong Huang, Shaohan Wang, Yiwei Yang, Tianchi Zhang, Zihan Huang, Haizhen Mei, Lingrui Fang, Junfeng Li, Zehao Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Liu, Shenghua Computation and Language Artificial Intelligence Information Retrieval Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at https://github.com/byronBBL/Context-DPO |
| title | Context-DPO: Aligning Language Models for Context-Faithfulness |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2412.15280 |