RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913859242033152 |
|---|---|
| author | Wang, Zihan Wang, Kangrui Wang, Qineng Zhang, Pingyue Li, Linjie Yang, Zhengyuan Jin, Xing Yu, Kefan Nguyen, Minh Nhat Liu, Licheng Gottlieb, Eli Lu, Yiping Cho, Kyunghyun Wu, Jiajun Fei-Fei, Li Wang, Lijuan Choi, Yejin Li, Manling |
| author_facet | Wang, Zihan Wang, Kangrui Wang, Qineng Zhang, Pingyue Li, Linjie Yang, Zhengyuan Jin, Xing Yu, Kefan Nguyen, Minh Nhat Liu, Licheng Gottlieb, Eli Lu, Yiping Cho, Kyunghyun Wu, Jiajun Fei-Fei, Li Wang, Lijuan Choi, Yejin Li, Manling |
| contents | Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20073 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning Wang, Zihan Wang, Kangrui Wang, Qineng Zhang, Pingyue Li, Linjie Yang, Zhengyuan Jin, Xing Yu, Kefan Nguyen, Minh Nhat Liu, Licheng Gottlieb, Eli Lu, Yiping Cho, Kyunghyun Wu, Jiajun Fei-Fei, Li Wang, Lijuan Choi, Yejin Li, Manling Machine Learning Artificial Intelligence Computation and Language Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variance cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts. Code and environments are available at https://github.com/RAGEN-AI/RAGEN. |
| title | RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2504.20073 |