Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912044849037312 |
|---|---|
| author | Xiong, Weimin Song, Yifan Zhao, Xiutian Wu, Wenhao Wang, Xun Wang, Ke Li, Cheng Peng, Wei Li, Sujian |
| author_facet | Xiong, Weimin Song, Yifan Zhao, Xiutian Wu, Wenhao Wang, Xun Wang, Ke Li, Cheng Peng, Wei Li, Sujian |
| contents | Large language model agents have exhibited exceptional performance across a range of complex interactive tasks. Recent approaches have utilized tuning with expert trajectories to enhance agent performance, yet they primarily concentrate on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. In this paper, we introduce the Iterative step-level Process Refinement (IPR) framework, which provides detailed step-by-step guidance to enhance agent training. Specifically, we adopt the Monte Carlo method to estimate step-level rewards. During each iteration, the agent explores along the expert trajectory and generates new actions. These actions are then evaluated against the corresponding step of expert trajectory using step-level rewards. Such comparison helps identify discrepancies, yielding contrastive action pairs that serve as training data for the agent. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines. Moreover, our analytical findings highlight the effectiveness of IPR in augmenting action efficiency and its applicability to diverse models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11176 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement Xiong, Weimin Song, Yifan Zhao, Xiutian Wu, Wenhao Wang, Xun Wang, Ke Li, Cheng Peng, Wei Li, Sujian Computation and Language Artificial Intelligence Machine Learning Large language model agents have exhibited exceptional performance across a range of complex interactive tasks. Recent approaches have utilized tuning with expert trajectories to enhance agent performance, yet they primarily concentrate on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. In this paper, we introduce the Iterative step-level Process Refinement (IPR) framework, which provides detailed step-by-step guidance to enhance agent training. Specifically, we adopt the Monte Carlo method to estimate step-level rewards. During each iteration, the agent explores along the expert trajectory and generates new actions. These actions are then evaluated against the corresponding step of expert trajectory using step-level rewards. Such comparison helps identify discrepancies, yielding contrastive action pairs that serve as training data for the agent. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines. Moreover, our analytical findings highlight the effectiveness of IPR in augmenting action efficiency and its applicability to diverse models. |
| title | Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.11176 |