Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.03817 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917862502825984 |
|---|---|
| author | Deng, Zhirui Dou, Zhicheng Zhu, Yutao Wen, Ji-Rong Xiong, Ruibin Wang, Mang Chen, Weipeng |
| author_facet | Deng, Zhirui Dou, Zhicheng Zhu, Yutao Wen, Ji-Rong Xiong, Ruibin Wang, Mang Chen, Weipeng |
| contents | The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_03817 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning Deng, Zhirui Dou, Zhicheng Zhu, Yutao Wen, Ji-Rong Xiong, Ruibin Wang, Mang Chen, Weipeng Artificial Intelligence Computation and Language Human-Computer Interaction Robotics The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods. |
| title | From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction Robotics |
| url | https://arxiv.org/abs/2411.03817 |