DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914619896889344 |
|---|---|
| author | Wang, Zhixin Xu, Jiaming Zhou, Tianyi Zhang, Mingjun Liu, Liming Hu, Jiarui Yang, Dian Wang, Tongyu Zhang, Ping Hou, Jinlong Feng, Siyuan Qi, Yuan Cheng, Yuan |
| author_facet | Wang, Zhixin Xu, Jiaming Zhou, Tianyi Zhang, Mingjun Liu, Liming Hu, Jiarui Yang, Dian Wang, Tongyu Zhang, Ping Hou, Jinlong Feng, Siyuan Qi, Yuan Cheng, Yuan |
| contents | Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DISTFLOW, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DISTFLOW establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DISTFLOW achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks. The source code is available at: https://github.com/sii-research/siiRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_13833 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training Wang, Zhixin Xu, Jiaming Zhou, Tianyi Zhang, Mingjun Liu, Liming Hu, Jiarui Yang, Dian Wang, Tongyu Zhang, Ping Hou, Jinlong Feng, Siyuan Qi, Yuan Cheng, Yuan Distributed, Parallel, and Cluster Computing Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DISTFLOW, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DISTFLOW establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DISTFLOW achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks. The source code is available at: https://github.com/sii-research/siiRL. |
| title | DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2507.13833 |