RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912791180345344 |
|---|---|
| author | Gao, Wei Zhao, Yuheng Wu, Tianyuan Xiong, Shaopan Wang, Weixun An, Dakai Cao, Lunxi Muhtar, Dilxat Liu, Zichen Zhao, Haizhou Huang, Ju Yang, Siran Li, Yongbin Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo Wang, Wei |
| author_facet | Gao, Wei Zhao, Yuheng Wu, Tianyuan Xiong, Shaopan Wang, Weixun An, Dakai Cao, Lunxi Muhtar, Dilxat Liu, Zichen Zhao, Haizhou Huang, Ju Yang, Siran Li, Yongbin Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo Wang, Wei |
| contents | Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning. Unlike standard LLM post-training, agentic RL workloads are highly heterogeneous, combining compute-intensive prefill phases, bandwidth-bound decoding, and stateful, CPU-heavy environment simulations. We argue that efficient agentic RL training requires disaggregated infrastructure to leverage specialized, best-fit hardware. However, naive disaggregation introduces substantial synchronization overhead and resource underutilization due to the complex dependencies between stages.
We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure. RollArc is built on three core principles: (1) hardware-affinity workload mapping, which routes compute-bound and bandwidth-bound tasks to bestfit GPU devices, (2) fine-grained asynchrony, which manages execution at the trajectory level to mitigate resource bubbles, and (3) statefulness-aware computation, which offloads stateless components (e.g., reward models) to serverless infrastructure for elastic scaling. Our results demonstrate that RollArc effectively improves training throughput and achieves 1.35-2.05\(\times\) end-to-end training time reduction compared to monolithic and synchronous baselines. We also evaluate RollArc by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with more than 3,000 GPUs, further demonstrating RollArc scalability and robustness. The code is available at https://github.com/alibaba/ROLL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22560 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure Gao, Wei Zhao, Yuheng Wu, Tianyuan Xiong, Shaopan Wang, Weixun An, Dakai Cao, Lunxi Muhtar, Dilxat Liu, Zichen Zhao, Haizhou Huang, Ju Yang, Siran Li, Yongbin Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo Wang, Wei Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning. Unlike standard LLM post-training, agentic RL workloads are highly heterogeneous, combining compute-intensive prefill phases, bandwidth-bound decoding, and stateful, CPU-heavy environment simulations. We argue that efficient agentic RL training requires disaggregated infrastructure to leverage specialized, best-fit hardware. However, naive disaggregation introduces substantial synchronization overhead and resource underutilization due to the complex dependencies between stages. We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure. RollArc is built on three core principles: (1) hardware-affinity workload mapping, which routes compute-bound and bandwidth-bound tasks to bestfit GPU devices, (2) fine-grained asynchrony, which manages execution at the trajectory level to mitigate resource bubbles, and (3) statefulness-aware computation, which offloads stateless components (e.g., reward models) to serverless infrastructure for elastic scaling. Our results demonstrate that RollArc effectively improves training throughput and achieves 1.35-2.05\(\times\) end-to-end training time reduction compared to monolithic and synchronous baselines. We also evaluate RollArc by training a hundreds-of-billions-parameter MoE model for Qoder product on an Alibaba cluster with more than 3,000 GPUs, further demonstrating RollArc scalability and robustness. The code is available at https://github.com/alibaba/ROLL. |
| title | RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.22560 |