Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2511.08325 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915612031188992 |
|---|---|
| author | Xi, Zhiheng Liao, Chenyang Li, Guanyu Yang, Yajie Chen, Wenxiang Zhang, Zhihao Wang, Binghai Jin, Senjie Zhou, Yuhao Guan, Jian Wu, Wei Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Xi, Zhiheng Liao, Chenyang Li, Guanyu Yang, Yajie Chen, Wenxiang Zhang, Zhihao Wang, Binghai Jin, Senjie Zhou, Yuhao Guan, Jian Wu, Wei Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Despite rapid development, large language models (LLMs) still encounter challenges in multi-turn decision-making tasks (i.e., agent tasks) like web shopping and browser navigation, which require making a sequence of intelligent decisions based on environmental feedback. Previous work for LLM agents typically relies on elaborate prompt engineering or fine-tuning with expert trajectories to improve performance. In this work, we take a different perspective: we explore constructing process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process. Unlike LLM reasoning, where each step is scored based on correctness, actions in agent tasks do not have a clear-cut correctness. Instead, they should be evaluated based on their proximity to the goal and the progress they have made. Building on this insight, we propose a re-defined PRM for agent tasks, named AgentPRM, to capture both the interdependence between sequential decisions and their contribution to the final goal. This enables better progress tracking and exploration-exploitation balance. To scalably obtain labeled data for training AgentPRM, we employ a Temporal Difference-based (TD-based) estimation method combined with Generalized Advantage Estimation (GAE), which proves more sample-efficient than prior methods. Extensive experiments across different agentic tasks show that AgentPRM is over $8\times$ more compute-efficient than baselines, and it demonstrates robust improvement when scaling up test-time compute. Moreover, we perform detailed analyses to show how our method works and offer more insights, e.g., applying AgentPRM to the reinforcement learning of LLM agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08325 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress Xi, Zhiheng Liao, Chenyang Li, Guanyu Yang, Yajie Chen, Wenxiang Zhang, Zhihao Wang, Binghai Jin, Senjie Zhou, Yuhao Guan, Jian Wu, Wei Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Information Retrieval Machine Learning Despite rapid development, large language models (LLMs) still encounter challenges in multi-turn decision-making tasks (i.e., agent tasks) like web shopping and browser navigation, which require making a sequence of intelligent decisions based on environmental feedback. Previous work for LLM agents typically relies on elaborate prompt engineering or fine-tuning with expert trajectories to improve performance. In this work, we take a different perspective: we explore constructing process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process. Unlike LLM reasoning, where each step is scored based on correctness, actions in agent tasks do not have a clear-cut correctness. Instead, they should be evaluated based on their proximity to the goal and the progress they have made. Building on this insight, we propose a re-defined PRM for agent tasks, named AgentPRM, to capture both the interdependence between sequential decisions and their contribution to the final goal. This enables better progress tracking and exploration-exploitation balance. To scalably obtain labeled data for training AgentPRM, we employ a Temporal Difference-based (TD-based) estimation method combined with Generalized Advantage Estimation (GAE), which proves more sample-efficient than prior methods. Extensive experiments across different agentic tasks show that AgentPRM is over $8\times$ more compute-efficient than baselines, and it demonstrates robust improvement when scaling up test-time compute. Moreover, we perform detailed analyses to show how our method works and offer more insights, e.g., applying AgentPRM to the reinforcement learning of LLM agents. |
| title | AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress |
| topic | Computation and Language Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2511.08325 |