Search Self-play: Pushing the Frontier of Agent Capability without Supervision

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lu, Hongliang, Wen, Yuhang, Cheng, Pengyu, Ding, Ruijin, Guo, Jiaqi, Xu, Haotian, Wang, Chutian, Chen, Haonan, Jiang, Xiaoxi, Jiang, Guanjun
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866916023582588928
author	Lu, Hongliang Wen, Yuhang Cheng, Pengyu Ding, Ruijin Guo, Jiaqi Xu, Haotian Wang, Chutian Chen, Haonan Jiang, Xiaoxi Jiang, Guanjun
author_facet	Lu, Hongliang Wen, Yuhang Cheng, Pengyu Ding, Ruijin Guo, Jiaqi Xu, Haotian Wang, Chutian Chen, Haonan Jiang, Xiaoxi Jiang, Guanjun
contents	Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.
format	Preprint
id	arxiv_https___arxiv_org_abs_2510_18821
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Search Self-play: Pushing the Frontier of Agent Capability without Supervision Lu, Hongliang Wen, Yuhang Cheng, Pengyu Ding, Ruijin Guo, Jiaqi Xu, Haotian Wang, Chutian Chen, Haonan Jiang, Xiaoxi Jiang, Guanjun Machine Learning Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.
title	Search Self-play: Pushing the Frontier of Agent Capability without Supervision
topic	Machine Learning
url	https://arxiv.org/abs/2510.18821