Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.13957 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911691176935424 |
|---|---|
| author | Xiong, Guangzhi Jin, Qiao Wang, Xiao Fang, Yin Liu, Haolin Yang, Yifan Chen, Fangyuan Song, Zhixing Wang, Dengyu Zhang, Minjia Lu, Zhiyong Zhang, Aidong |
| author_facet | Xiong, Guangzhi Jin, Qiao Wang, Xiao Fang, Yin Liu, Haolin Yang, Yifan Chen, Fangyuan Song, Zhixing Wang, Dengyu Zhang, Minjia Lu, Zhiyong Zhang, Aidong |
| contents | Large language models (LLMs) are transforming web search by shifting from document ranking to synthesizing answers, and are increasingly deployed as autonomous agentic search systems that iteratively interact with external knowledge sources. Despite this progress, building effective search agents remains challenging because high-quality intermediate search steps are difficult to generate. Previous approaches have primarily relied on outcome supervision, rewarding agents only for producing correct final answers. This often leads to reward hacking and excessive dependence on parametric memory, limiting generalization to out-of-domain tasks. To address these limitations, we introduce RAG-Gym, a framework that shifts supervision from final answers to the search process itself. With RAG-Gym, we systematically investigate architecture design, parameter optimization, and action evaluation, identifying reasoning reflection as a critical capability for search agents. Building on this insight, we propose Re$^2$Search++, a process-supervised agent that achieves substantial improvements on multi-hop information-seeking benchmarks, especially in out-of-domain settings. Performance gains are driven primarily by higher-quality search queries rather than answer optimization alone, and the learned search critics transfer across models, including proprietary LLMs. These findings show that supervising the search process produces more reliable and generalizable information-seeking agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13957 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Supervising the search process produces reliable and generalizable information-seeking agents Xiong, Guangzhi Jin, Qiao Wang, Xiao Fang, Yin Liu, Haolin Yang, Yifan Chen, Fangyuan Song, Zhixing Wang, Dengyu Zhang, Minjia Lu, Zhiyong Zhang, Aidong Computation and Language Artificial Intelligence Large language models (LLMs) are transforming web search by shifting from document ranking to synthesizing answers, and are increasingly deployed as autonomous agentic search systems that iteratively interact with external knowledge sources. Despite this progress, building effective search agents remains challenging because high-quality intermediate search steps are difficult to generate. Previous approaches have primarily relied on outcome supervision, rewarding agents only for producing correct final answers. This often leads to reward hacking and excessive dependence on parametric memory, limiting generalization to out-of-domain tasks. To address these limitations, we introduce RAG-Gym, a framework that shifts supervision from final answers to the search process itself. With RAG-Gym, we systematically investigate architecture design, parameter optimization, and action evaluation, identifying reasoning reflection as a critical capability for search agents. Building on this insight, we propose Re$^2$Search++, a process-supervised agent that achieves substantial improvements on multi-hop information-seeking benchmarks, especially in out-of-domain settings. Performance gains are driven primarily by higher-quality search queries rather than answer optimization alone, and the learned search critics transfer across models, including proprietary LLMs. These findings show that supervising the search process produces more reliable and generalizable information-seeking agents. |
| title | Supervising the search process produces reliable and generalizable information-seeking agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.13957 |