Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912702878711808 |
|---|---|
| author | Si, Shuzheng Zhao, Haozhe Gao, Cheng Bai, Yuzhuo Wang, Zhitong Gao, Bofei Luo, Kangyang Li, Wenhao Huang, Yufei Chen, Gang Qi, Fanchao Zhang, Minjia Chang, Baobao Sun, Maosong |
| author_facet | Si, Shuzheng Zhao, Haozhe Gao, Cheng Bai, Yuzhuo Wang, Zhitong Gao, Bofei Luo, Kangyang Li, Wenhao Huang, Yufei Chen, Gang Qi, Fanchao Zhang, Minjia Chang, Baobao Sun, Maosong |
| contents | Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning Si, Shuzheng Zhao, Haozhe Gao, Cheng Bai, Yuzhuo Wang, Zhitong Gao, Bofei Luo, Kangyang Li, Wenhao Huang, Yufei Chen, Gang Qi, Fanchao Zhang, Minjia Chang, Baobao Sun, Maosong Computation and Language Artificial Intelligence Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1. |
| title | Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.16483 |