Exploring Reasoning Reward Model for Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918470444122112 |
|---|---|
| author | Fan, Kaixuan Feng, Kaituo Zhang, Manyuan Peng, Tianshuo Li, Zhixun Jiang, Yilei Chen, Shuang Pei, Peng Cai, Xunliang Yue, Xiangyu |
| author_facet | Fan, Kaixuan Feng, Kaituo Zhang, Manyuan Peng, Tianshuo Li, Zhixun Jiang, Yilei Chen, Shuang Pei, Peng Cai, Xunliang Yue, Xiangyu |
| contents | Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22154 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exploring Reasoning Reward Model for Agents Fan, Kaixuan Feng, Kaituo Zhang, Manyuan Peng, Tianshuo Li, Zhixun Jiang, Yilei Chen, Shuang Pei, Peng Cai, Xunliang Yue, Xiangyu Artificial Intelligence Computation and Language Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research. |
| title | Exploring Reasoning Reward Model for Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2601.22154 |