Improving Retrospective Language Agents via Joint Policy Gradient Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912256788267008 |
|---|---|
| author | Feng, Xueyang Lan, Bo Dai, Quanyu Wang, Lei Tang, Jiakai Chen, Xu Dong, Zhenhua Wen, Ji-Rong |
| author_facet | Feng, Xueyang Lan, Bo Dai, Quanyu Wang, Lei Tang, Jiakai Chen, Xu Dong, Zhenhua Wen, Ji-Rong |
| contents | In recent research advancements within the community, large language models (LLMs) have sparked great interest in creating autonomous agents. However, current prompt-based agents often heavily rely on large-scale LLMs. Meanwhile, although fine-tuning methods significantly enhance the capabilities of smaller LLMs, the fine-tuned agents often lack the potential for self-reflection and self-improvement. To address these challenges, we introduce a novel agent framework named RetroAct, which is a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents. Specifically, we develop a two-stage joint optimization process that integrates imitation learning and reinforcement learning, and design an off-policy joint policy gradient optimization algorithm with imitation learning regularization to enhance the data efficiency and training stability in agent tasks. RetroAct significantly improves the performance of open-source models, reduces dependency on closed-source LLMs, and enables fine-tuned agents to learn and evolve continuously. We conduct extensive experiments across various testing environments, demonstrating RetroAct has substantial improvements in task performance and decision-making processes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01490 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Retrospective Language Agents via Joint Policy Gradient Optimization Feng, Xueyang Lan, Bo Dai, Quanyu Wang, Lei Tang, Jiakai Chen, Xu Dong, Zhenhua Wen, Ji-Rong Computation and Language I.2.7 In recent research advancements within the community, large language models (LLMs) have sparked great interest in creating autonomous agents. However, current prompt-based agents often heavily rely on large-scale LLMs. Meanwhile, although fine-tuning methods significantly enhance the capabilities of smaller LLMs, the fine-tuned agents often lack the potential for self-reflection and self-improvement. To address these challenges, we introduce a novel agent framework named RetroAct, which is a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents. Specifically, we develop a two-stage joint optimization process that integrates imitation learning and reinforcement learning, and design an off-policy joint policy gradient optimization algorithm with imitation learning regularization to enhance the data efficiency and training stability in agent tasks. RetroAct significantly improves the performance of open-source models, reduces dependency on closed-source LLMs, and enables fine-tuned agents to learn and evolve continuously. We conduct extensive experiments across various testing environments, demonstrating RetroAct has substantial improvements in task performance and decision-making processes. |
| title | Improving Retrospective Language Agents via Joint Policy Gradient Optimization |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2503.01490 |