Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913018567196672 |
|---|---|
| author | Hao, Chuzhan Feng, Wenfeng Jiang, Guochao Quan, Guofeng Liu, Guohua Zhang, Yuewei |
| author_facet | Hao, Chuzhan Feng, Wenfeng Jiang, Guochao Quan, Guofeng Liu, Guohua Zhang, Yuewei |
| contents | Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08124 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search Hao, Chuzhan Feng, Wenfeng Jiang, Guochao Quan, Guofeng Liu, Guohua Zhang, Yuewei Artificial Intelligence Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization. |
| title | Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.08124 |