LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913855025709056 |
|---|---|
| author | Zhang, Qi Yang, Shouqing Gao, Lirong Chen, Hao Hu, Xiaomeng Chen, Jinglei Wang, Jiexiang Guo, Sheng Zheng, Bo Wang, Haobo Zhao, Junbo |
| author_facet | Zhang, Qi Yang, Shouqing Gao, Lirong Chen, Hao Hu, Xiaomeng Chen, Jinglei Wang, Jiexiang Guo, Sheng Zheng, Bo Wang, Haobo Zhao, Junbo |
| contents | Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17447 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization Zhang, Qi Yang, Shouqing Gao, Lirong Chen, Hao Hu, Xiaomeng Chen, Jinglei Wang, Jiexiang Guo, Sheng Zheng, Bo Wang, Haobo Zhao, Junbo Computation and Language Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon. |
| title | LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.17447 |