How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xu, Yinuo, Lu, Shuo, Cheng, Jianjie, Wang, Meng, Xie, Qianlong, Wang, Xingxing, He, Ran, Liang, Jian
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866911462828539904
author	Xu, Yinuo Lu, Shuo Cheng, Jianjie Wang, Meng Xie, Qianlong Wang, Xingxing He, Ran Liang, Jian
author_facet	Xu, Yinuo Lu, Shuo Cheng, Jianjie Wang, Meng Xie, Qianlong Wang, Xingxing He, Ran Liang, Jian
contents	Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.
format	Preprint
id	arxiv_https___arxiv_org_abs_2602_19526
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1 Xu, Yinuo Lu, Shuo Cheng, Jianjie Wang, Meng Xie, Qianlong Wang, Xingxing He, Ran Liang, Jian Computation and Language Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.
title	How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
topic	Computation and Language
url	https://arxiv.org/abs/2602.19526