Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914608267132928 |
|---|---|
| author | Wu, Jiayi Xie, Ruobing Huang, Zeqian Jiang, Lei Xu, Can Luo, Kangyang Lin, Bochen Gao, Ming Li, Xiang |
| author_facet | Wu, Jiayi Xie, Ruobing Huang, Zeqian Jiang, Lei Xu, Can Luo, Kangyang Lin, Bochen Gao, Ming Li, Xiang |
| contents | Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used training algorithm. However, GRPO-style algorithms still face several challenges in multi-hop search settings. First, correct intermediate steps are often penalized when the final answer is wrong. Second, training is highly unstable, often causing degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for search agents that enables more accurate and more stable modeling of penalties and rewards. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then further rebalances positive and negative advantages to improve training stability. Importantly, CalibAdv adopts a lightweight design that calibrates advantages from standard rollout signals, making it simple and easy to deploy. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at https://github.com/wujwyi/CalibAdv. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18235 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents Wu, Jiayi Xie, Ruobing Huang, Zeqian Jiang, Lei Xu, Can Luo, Kangyang Lin, Bochen Gao, Ming Li, Xiang Computation and Language Artificial Intelligence Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used training algorithm. However, GRPO-style algorithms still face several challenges in multi-hop search settings. First, correct intermediate steps are often penalized when the final answer is wrong. Second, training is highly unstable, often causing degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for search agents that enables more accurate and more stable modeling of penalties and rewards. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then further rebalances positive and negative advantages to improve training stability. Importantly, CalibAdv adopts a lightweight design that calibrates advantages from standard rollout signals, making it simple and easy to deploy. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at https://github.com/wujwyi/CalibAdv. |
| title | Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.18235 |