AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914222110146560 |
|---|---|
| author | Cai, Shihao Fang, Runnan Wu, Jialong Li, Baixuan Wang, Xinyu Jiang, Yong Su, Liangcai Zhang, Liwen Yin, Wenbiao Zhang, Zhen Feng, Fuli Xie, Pengjun Wang, Xiaobin |
| author_facet | Cai, Shihao Fang, Runnan Wu, Jialong Li, Baixuan Wang, Xinyu Jiang, Yong Su, Liangcai Zhang, Liwen Yin, Wenbiao Zhang, Zhen Feng, Fuli Xie, Pengjun Wang, Xiaobin |
| contents | Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22857 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning Cai, Shihao Fang, Runnan Wu, Jialong Li, Baixuan Wang, Xinyu Jiang, Yong Su, Liangcai Zhang, Liwen Yin, Wenbiao Zhang, Zhen Feng, Fuli Xie, Pengjun Wang, Xiaobin Computation and Language Artificial Intelligence Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization. |
| title | AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.22857 |