Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2508.05387 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912534118793216 |
|---|---|
| author | Xiao, Jie Fan, Changyuan Ren, Qingnan Long, Alfred Zhang, Yuchen Yu, Rymon Yang, Eric Ai, Lynn Gan, Shaoduo |
| author_facet | Xiao, Jie Fan, Changyuan Ren, Qingnan Long, Alfred Zhang, Yuchen Yu, Rymon Yang, Eric Ai, Lynn Gan, Shaoduo |
| contents | Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between inference and training workloads. This serial context switching violates the single-program-multiple-data (SPMD) assumption underlying today's distributed training systems. We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms while preserving statistical efficiency. Echo introduces two lightweight synchronization protocols: a sequential pull mode that refreshes policy weights according to API call for minimal bias, and an asynchronous push-pull mode that streams version-tagged rollouts through a replay buffer to maximise hardware utilisation. Training four representative RL workloads with Qwen3-4B, Qwen2.5-7B, Qwen3-30B-A3B-Thinking-2507 and Qwen3-32B on a geographically distributed cluster, Echo matches a fully co-located Verl baseline in convergence speed and final reward while off-loading trajectory generation to commodity edge hardware. These promising results demonstrate that large-scale RL for LLMs could achieve datacentre-grade performance using decentralised, heterogeneous resources. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05387 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms Xiao, Jie Fan, Changyuan Ren, Qingnan Long, Alfred Zhang, Yuchen Yu, Rymon Yang, Eric Ai, Lynn Gan, Shaoduo Machine Learning Artificial Intelligence Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between inference and training workloads. This serial context switching violates the single-program-multiple-data (SPMD) assumption underlying today's distributed training systems. We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms while preserving statistical efficiency. Echo introduces two lightweight synchronization protocols: a sequential pull mode that refreshes policy weights according to API call for minimal bias, and an asynchronous push-pull mode that streams version-tagged rollouts through a replay buffer to maximise hardware utilisation. Training four representative RL workloads with Qwen3-4B, Qwen2.5-7B, Qwen3-30B-A3B-Thinking-2507 and Qwen3-32B on a geographically distributed cluster, Echo matches a fully co-located Verl baseline in convergence speed and final reward while off-loading trajectory generation to commodity edge hardware. These promising results demonstrate that large-scale RL for LLMs could achieve datacentre-grade performance using decentralised, heterogeneous resources. |
| title | Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.05387 |