Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2510.13272 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914304731643904 |
|---|---|
| author | Xu, Zhichao Wu, Zongyu Zhou, Yun Feng, Aosong Zhou, Kang Woo, Sangmin Ramnath, Kiran Tian, Yijun Qi, Xuan Qiu, Weikang Cheong, Lin Lee Ding, Haibo |
| author_facet | Xu, Zhichao Wu, Zongyu Zhou, Yun Feng, Aosong Zhou, Kang Woo, Sangmin Ramnath, Kiran Tian, Yijun Qi, Xuan Qiu, Weikang Cheong, Lin Lee Ding, Haibo |
| contents | Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent works have begun exploring how to train LLMs to use search engines more effectively as tools for retrieval-augmented generation. Although these methods achieve performance improvement across QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for evaluating RL-based search agents, covering three distinct faithfulness metrics: information-think faithfulness, think-answer faithfulness, and think-search faithfulness. Our evaluations reveal that canonical search agents trained via Reinforcement Learning from Verifiable Reward (RLVR) -- including SearchR1 and ReSearch -- have significant room for improvement in this regard. To foster faithful reasoning, we introduce VERITAS(Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with VERITAS not only significantly improve reasoning faithfulness, but also achieve better task performance compared to the baselines trained against pure outcome-based reward. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13272 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation Xu, Zhichao Wu, Zongyu Zhou, Yun Feng, Aosong Zhou, Kang Woo, Sangmin Ramnath, Kiran Tian, Yijun Qi, Xuan Qiu, Weikang Cheong, Lin Lee Ding, Haibo Computation and Language Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent works have begun exploring how to train LLMs to use search engines more effectively as tools for retrieval-augmented generation. Although these methods achieve performance improvement across QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for evaluating RL-based search agents, covering three distinct faithfulness metrics: information-think faithfulness, think-answer faithfulness, and think-search faithfulness. Our evaluations reveal that canonical search agents trained via Reinforcement Learning from Verifiable Reward (RLVR) -- including SearchR1 and ReSearch -- have significant room for improvement in this regard. To foster faithful reasoning, we introduce VERITAS(Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with VERITAS not only significantly improve reasoning faithfulness, but also achieve better task performance compared to the baselines trained against pure outcome-based reward. |
| title | Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.13272 |