Improving Search Agent with One Line of Code

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Jian, Chen, Dongsheng, Xu, Zhenhua, Jin, Yizhang, Wu, Jiafu, Wang, Chengjie, Yuan, Xiaotong, Wang, Yabiao
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866915851632902144
author	Li, Jian Chen, Dongsheng Xu, Zhenhua Jin, Yizhang Wu, Jiafu Wang, Chengjie Yuan, Xiaotong Wang, Yabiao
author_facet	Li, Jian Chen, Dongsheng Xu, Zhenhua Jin, Yizhang Wu, Jiafu Wang, Chengjie Yuan, Xiaotong Wang, Yabiao
contents	Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools for a multi-turn information-seeking process autonomously. However, we identify a critical training instability that leads to catastrophic model collapse: Importance Sampling Distribution Drift(ISDD). In Group Relative Policy Optimization(GRPO), a widely adopted TARL algorithm, ISDD manifests as a precipitous decline in the importance sampling ratios, which nullifies gradient updates and triggers irreversible training failure. To address this, we propose \textbf{S}earch \textbf{A}gent \textbf{P}olicy \textbf{O}ptimization (\textbf{SAPO}), which stabilizes training via a conditional token-level KL constraint. Unlike hard clipping, which ignores distributional divergence, SAPO selectively penalizes the KL divergence between the current and old policies. Crucially, this penalty is applied only to positive tokens with low probabilities where the policy has shifted excessively, thereby preventing distribution drift while preserving gradient flow. Remarkably, SAPO requires only one-line code modification to standard GRPO, ensuring immediate deployability. Extensive experiments across seven QA benchmarks demonstrate that SAPO achieves \textbf{+10.6\% absolute improvement} (+31.5\% relative) over Search-R1, yielding consistent gains across varying model scales (1.5B, 14B) and families (Qwen, LLaMA).
format	Preprint
id	arxiv_https___arxiv_org_abs_2603_10069
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	Improving Search Agent with One Line of Code Li, Jian Chen, Dongsheng Xu, Zhenhua Jin, Yizhang Wu, Jiafu Wang, Chengjie Yuan, Xiaotong Wang, Yabiao Machine Learning Computation and Language Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools for a multi-turn information-seeking process autonomously. However, we identify a critical training instability that leads to catastrophic model collapse: Importance Sampling Distribution Drift(ISDD). In Group Relative Policy Optimization(GRPO), a widely adopted TARL algorithm, ISDD manifests as a precipitous decline in the importance sampling ratios, which nullifies gradient updates and triggers irreversible training failure. To address this, we propose \textbf{S}earch \textbf{A}gent \textbf{P}olicy \textbf{O}ptimization (\textbf{SAPO}), which stabilizes training via a conditional token-level KL constraint. Unlike hard clipping, which ignores distributional divergence, SAPO selectively penalizes the KL divergence between the current and old policies. Crucially, this penalty is applied only to positive tokens with low probabilities where the policy has shifted excessively, thereby preventing distribution drift while preserving gradient flow. Remarkably, SAPO requires only one-line code modification to standard GRPO, ensuring immediate deployability. Extensive experiments across seven QA benchmarks demonstrate that SAPO achieves \textbf{+10.6\% absolute improvement} (+31.5\% relative) over Search-R1, yielding consistent gains across varying model scales (1.5B, 14B) and families (Qwen, LLaMA).
title	Improving Search Agent with One Line of Code
topic	Machine Learning Computation and Language
url	https://arxiv.org/abs/2603.10069