Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918047588024320 |
|---|---|
| author | Wang, Weixun Xiong, Shaopan Chen, Gengru Gao, Wei Guo, Sheng He, Yancheng Huang, Ju Liu, Jiaheng Li, Zhendong Li, Xiaoyang Liu, Zichen Zhao, Haizhou An, Dakai Cao, Lunxi Cao, Qiyang Deng, Wanxi Du, Feilei Gu, Yiliang Li, Jiahe Li, Xiang Liu, Mingjie Luo, Yijia Liu, Zihe Wang, Yadao Wang, Pei Wu, Tianyuan Wu, Yanan Zhao, Yuheng Zhao, Shuaibing Yang, Jin Yang, Siran Tan, Yingshui Yi, Huimin Xu, Yuchi Yuan, Yujin Zhang, Xingyao Qu, Lin Su, Wenbo Wang, Wei Wang, Jiamang Zheng, Bo |
| author_facet | Wang, Weixun Xiong, Shaopan Chen, Gengru Gao, Wei Guo, Sheng He, Yancheng Huang, Ju Liu, Jiaheng Li, Zhendong Li, Xiaoyang Liu, Zichen Zhao, Haizhou An, Dakai Cao, Lunxi Cao, Qiyang Deng, Wanxi Du, Feilei Gu, Yiliang Li, Jiahe Li, Xiang Liu, Mingjie Luo, Yijia Liu, Zihe Wang, Yadao Wang, Pei Wu, Tianyuan Wu, Yanan Zhao, Yuheng Zhao, Shuaibing Yang, Jin Yang, Siran Tan, Yingshui Yi, Huimin Xu, Yuchi Yuan, Yujin Zhang, Xingyao Qu, Lin Su, Wenbo Wang, Wei Wang, Jiamang Zheng, Bo |
| contents | We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user groups: tech pioneers aiming for cost-effective, fault-tolerant large-scale training, developers requiring flexible control over training workflows, and researchers seeking agile experimentation. ROLL is built upon several key modules to serve these user groups effectively. First, a single-controller architecture combined with an abstraction of the parallel worker simplifies the development of the training pipeline. Second, the parallel strategy and data transfer modules enable efficient and scalable training. Third, the rollout scheduler offers fine-grained management of each sample's lifecycle during the rollout stage. Fourth, the environment worker and reward worker support rapid and flexible experimentation with agentic RL algorithms and reward designs. Finally, AutoDeviceMapping allows users to assign resources to different models flexibly across various stages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06122 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library Wang, Weixun Xiong, Shaopan Chen, Gengru Gao, Wei Guo, Sheng He, Yancheng Huang, Ju Liu, Jiaheng Li, Zhendong Li, Xiaoyang Liu, Zichen Zhao, Haizhou An, Dakai Cao, Lunxi Cao, Qiyang Deng, Wanxi Du, Feilei Gu, Yiliang Li, Jiahe Li, Xiang Liu, Mingjie Luo, Yijia Liu, Zihe Wang, Yadao Wang, Pei Wu, Tianyuan Wu, Yanan Zhao, Yuheng Zhao, Shuaibing Yang, Jin Yang, Siran Tan, Yingshui Yi, Huimin Xu, Yuchi Yuan, Yujin Zhang, Xingyao Qu, Lin Su, Wenbo Wang, Wei Wang, Jiamang Zheng, Bo Machine Learning Distributed, Parallel, and Cluster Computing We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user groups: tech pioneers aiming for cost-effective, fault-tolerant large-scale training, developers requiring flexible control over training workflows, and researchers seeking agile experimentation. ROLL is built upon several key modules to serve these user groups effectively. First, a single-controller architecture combined with an abstraction of the parallel worker simplifies the development of the training pipeline. Second, the parallel strategy and data transfer modules enable efficient and scalable training. Third, the rollout scheduler offers fine-grained management of each sample's lifecycle during the rollout stage. Fourth, the environment worker and reward worker support rapid and flexible experimentation with agentic RL algorithms and reward designs. Finally, AutoDeviceMapping allows users to assign resources to different models flexibly across various stages. |
| title | Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2506.06122 |