Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.13019 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908884255375360 |
|---|---|
| author | Xiao, Bangjun Zhao, Yihao Deng, Xiangwei Yu, Shihua Xiang, Yuxing Liu, Huaqiu Wang, Qiying Zhao, Liang Zhang, Hailin Liu, Xuanzhe Jin, Xin Luo, Fuli |
| author_facet | Xiao, Bangjun Zhao, Yihao Deng, Xiangwei Yu, Shihua Xiang, Yuxing Liu, Huaqiu Wang, Qiying Zhao, Liang Zhang, Hailin Liu, Xuanzhe Jin, Xin Luo, Fuli |
| contents | Agentic reinforcement learning (RL) has emerged as a transformative workload in cloud clusters, enabling large language models (LLMs) to solve complex problems through interactions with real world. However, unlike traditional RL, agentic RL demands substantial external cloud resources, e.g., CPUs for code execution and GPUs for reward models, that exist outside the primary training cluster. Existing agentic RL framework typically rely on static over-provisioning, i.e., resources are often tied to long-lived trajectories or isolated by tasks, which leads to severe resource inefficiency.
We propose the action-level orchestration, and incorporate it into ARL-Tangram, a unified resource management system that enables fine-grained external resource sharing and elasticity. ARL-Tangram utilizes a unified action-level formulation and an elastic scheduling algorithm to minimize action completion time (ACT) while satisfying heterogeneous resource constraints. Further, heterogeneous resource managers are tailored to efficiently support the action-level execution on resources with heterogeneous characteristics and topologies. Evaluation on real-world agentic RL tasks demonstrates that ARL-Tangram improves average ACT by up to 4.3$\times$, speeds up the step duration of RL training by up to 1.5$\times$, and saves the external resources by up to 71.2$\%$. This system has been deployed to support the training of the MiMo series models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13019 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning Xiao, Bangjun Zhao, Yihao Deng, Xiangwei Yu, Shihua Xiang, Yuxing Liu, Huaqiu Wang, Qiying Zhao, Liang Zhang, Hailin Liu, Xuanzhe Jin, Xin Luo, Fuli Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Agentic reinforcement learning (RL) has emerged as a transformative workload in cloud clusters, enabling large language models (LLMs) to solve complex problems through interactions with real world. However, unlike traditional RL, agentic RL demands substantial external cloud resources, e.g., CPUs for code execution and GPUs for reward models, that exist outside the primary training cluster. Existing agentic RL framework typically rely on static over-provisioning, i.e., resources are often tied to long-lived trajectories or isolated by tasks, which leads to severe resource inefficiency. We propose the action-level orchestration, and incorporate it into ARL-Tangram, a unified resource management system that enables fine-grained external resource sharing and elasticity. ARL-Tangram utilizes a unified action-level formulation and an elastic scheduling algorithm to minimize action completion time (ACT) while satisfying heterogeneous resource constraints. Further, heterogeneous resource managers are tailored to efficiently support the action-level execution on resources with heterogeneous characteristics and topologies. Evaluation on real-world agentic RL tasks demonstrates that ARL-Tangram improves average ACT by up to 4.3$\times$, speeds up the step duration of RL training by up to 1.5$\times$, and saves the external resources by up to 71.2$\%$. This system has been deployed to support the training of the MiMo series models. |
| title | ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.13019 |