Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2507.19017 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912501012103168 |
|---|---|
| author | Feng, Laingjun Pan, Chenyi Guo, Xinjie Mei, Fei Ning, Benzhe Zhang, Jianxiang Liu, Xinyang Zhou, Beirong Shu, Zeng Liu, Chang Yang, Guang Han, Zhenyu Wang, Jiangben Wang, Bo |
| author_facet | Feng, Laingjun Pan, Chenyi Guo, Xinjie Mei, Fei Ning, Benzhe Zhang, Jianxiang Liu, Xinyang Zhou, Beirong Shu, Zeng Liu, Chang Yang, Guang Han, Zhenyu Wang, Jiangben Wang, Bo |
| contents | Reinforcement learning (RL) is a paradigm increasingly used to align large language models. Popular RL algorithms utilize multiple workers and can be modeled as a graph, where each node is the status of a worker and each edge represents dataflow between nodes. Owing to the heavy cross-node dependencies, the RL training system usually suffers from poor cluster scalability and low memory utilization. In this article, we introduce MindSpeed RL, an effective and efficient system for large-scale RL training. Unlike existing centralized methods, MindSpeed RL organizes the essential data dependencies in RL training, i.e., sample flow and resharding flow, from a distributed view. On the one hand, a distributed transfer dock strategy, which sets controllers and warehouses on the basis of the conventional replay buffer, is designed to release the dispatch overhead in the sample flow. A practical allgather--swap strategy is presented to eliminate redundant memory usage in resharding flow. In addition, MindSpeed RL further integrates numerous parallelization strategies and acceleration techniques for systematic optimization. Compared with existing state-of-the-art systems, comprehensive experiments on the RL training of popular Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B show that MindSpeed RL increases the throughput by 1.42 ~ 3.97 times. Finally, we open--source MindSpeed RL and perform all the experiments on a super pod of Ascend with 384 neural processing units (NPUs) to demonstrate the powerful performance and reliability of Ascend. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19017 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster Feng, Laingjun Pan, Chenyi Guo, Xinjie Mei, Fei Ning, Benzhe Zhang, Jianxiang Liu, Xinyang Zhou, Beirong Shu, Zeng Liu, Chang Yang, Guang Han, Zhenyu Wang, Jiangben Wang, Bo Machine Learning Artificial Intelligence CS Reinforcement learning (RL) is a paradigm increasingly used to align large language models. Popular RL algorithms utilize multiple workers and can be modeled as a graph, where each node is the status of a worker and each edge represents dataflow between nodes. Owing to the heavy cross-node dependencies, the RL training system usually suffers from poor cluster scalability and low memory utilization. In this article, we introduce MindSpeed RL, an effective and efficient system for large-scale RL training. Unlike existing centralized methods, MindSpeed RL organizes the essential data dependencies in RL training, i.e., sample flow and resharding flow, from a distributed view. On the one hand, a distributed transfer dock strategy, which sets controllers and warehouses on the basis of the conventional replay buffer, is designed to release the dispatch overhead in the sample flow. A practical allgather--swap strategy is presented to eliminate redundant memory usage in resharding flow. In addition, MindSpeed RL further integrates numerous parallelization strategies and acceleration techniques for systematic optimization. Compared with existing state-of-the-art systems, comprehensive experiments on the RL training of popular Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B show that MindSpeed RL increases the throughput by 1.42 ~ 3.97 times. Finally, we open--source MindSpeed RL and perform all the experiments on a super pod of Ascend with 384 neural processing units (NPUs) to demonstrate the powerful performance and reliability of Ascend. |
| title | MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster |
| topic | Machine Learning Artificial Intelligence CS |
| url | https://arxiv.org/abs/2507.19017 |