SSRL: Self-Search Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916899626942464 |
|---|---|
| author | Fan, Yuchen Zhang, Kaiyan Zhou, Heng Zuo, Yuxin Chen, Yanxu Fu, Yu Long, Xinwei Zhu, Xuekai Jiang, Che Zhang, Yuchen Kang, Li Chen, Gang Huang, Cheng He, Zhizhou Wang, Bingning Bai, Lei Ding, Ning Zhou, Bowen |
| author_facet | Fan, Yuchen Zhang, Kaiyan Zhou, Heng Zuo, Yuxin Chen, Yanxu Fu, Yu Long, Xinwei Zhu, Xuekai Jiang, Che Zhang, Yuchen Kang, Li Chen, Gang Huang, Cheng He, Zhizhou Wang, Bingning Bai, Lei Ding, Ning Zhou, Bowen |
| contents | We investigate the potential of large language models (LLMs) to serve as efficient simulators for agentic search tasks in reinforcement learning (RL), thereby reducing dependence on costly interactions with external search engines. To this end, we first quantify the intrinsic search capability of LLMs via structured prompting and repeated sampling, which we term Self-Search. Our results reveal that LLMs exhibit strong scaling behavior with respect to the inference budget, achieving high pass@k on question-answering benchmarks, including the challenging BrowseComp task. Building on these observations, we introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability through format-based and rule-based rewards. SSRL enables models to iteratively refine their knowledge utilization internally, without requiring access to external tools. Empirical evaluations demonstrate that SSRL-trained policy models provide a cost-effective and stable environment for search-driven RL training, reducing reliance on external search engines and facilitating robust sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world knowledge that can be effectively elicited to achieve high performance; 2) SSRL demonstrates the potential of leveraging internal knowledge to reduce hallucination; 3) SSRL-trained models integrate seamlessly with external search engines without additional effort. Our findings highlight the potential of LLMs to support more scalable RL agent training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10874 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SSRL: Self-Search Reinforcement Learning Fan, Yuchen Zhang, Kaiyan Zhou, Heng Zuo, Yuxin Chen, Yanxu Fu, Yu Long, Xinwei Zhu, Xuekai Jiang, Che Zhang, Yuchen Kang, Li Chen, Gang Huang, Cheng He, Zhizhou Wang, Bingning Bai, Lei Ding, Ning Zhou, Bowen Computation and Language We investigate the potential of large language models (LLMs) to serve as efficient simulators for agentic search tasks in reinforcement learning (RL), thereby reducing dependence on costly interactions with external search engines. To this end, we first quantify the intrinsic search capability of LLMs via structured prompting and repeated sampling, which we term Self-Search. Our results reveal that LLMs exhibit strong scaling behavior with respect to the inference budget, achieving high pass@k on question-answering benchmarks, including the challenging BrowseComp task. Building on these observations, we introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability through format-based and rule-based rewards. SSRL enables models to iteratively refine their knowledge utilization internally, without requiring access to external tools. Empirical evaluations demonstrate that SSRL-trained policy models provide a cost-effective and stable environment for search-driven RL training, reducing reliance on external search engines and facilitating robust sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world knowledge that can be effectively elicited to achieve high performance; 2) SSRL demonstrates the potential of leveraging internal knowledge to reduce hallucination; 3) SSRL-trained models integrate seamlessly with external search engines without additional effort. Our findings highlight the potential of LLMs to support more scalable RL agent training. |
| title | SSRL: Self-Search Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.10874 |