Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.26256 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918473011036160 |
|---|---|
| author | Hu, Tianhao Liu, Xiangcheng Xiao, Youshao Zheng, Yang Huang, Xuan Ding, Jinrui Zhang, Yufei Liang, Tao Zang, Hongyu Chen, Quan Sun, Yueqing Shi, Wenjie Zhang, Chao Wang, Wei Gu, Qi Sun, Yerui Xie, Yucheng Cai, Xunliang |
| author_facet | Hu, Tianhao Liu, Xiangcheng Xiao, Youshao Zheng, Yang Huang, Xuan Ding, Jinrui Zhang, Yufei Liang, Tao Zang, Hongyu Chen, Quan Sun, Yueqing Shi, Wenjie Zhang, Chao Wang, Wei Gu, Qi Sun, Yerui Xie, Yucheng Cai, Xunliang |
| contents | Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline. Asynchronous training offers a natural remedy by overlapping generation with training, but introduces a fundamental tension between efficiency and algorithmic correctness. We identify three constraints in asynchronous training to preserve convergence: intra-trajectory policy consistency, data integrity, and bounded staleness. Existing approaches fail to intrinsically address the long-tailed trajectory problem, which is further exacerbated by the imbalance characteristic of Mix-of-Experts models, or deviate from the standard RL training formulation, thereby hindering model convergence. Therefore, we propose DORA (Dynamic ORchestration for Asynchronous Rollout), which addresses this challenge through algorithm-system co-design. DORA introduces multi-version streaming rollout, a novel asynchronous paradigm that maintains multiple policy versions concurrently -- simultaneously achieving full bubble elimination without compromising algorithmic constraints. Experimental results demonstrate that our DORA system achieves substantial improvements in throughput -- up to 2--3 times higher than state-of-the-art systems on open-source benchmarks -- without compromising convergence. Furthermore, in large-scale industrial applications with tens of thousands of accelerators, DORA accelerates RL training by 2--4 times compared to synchronous training across various scenarios. The resultant open-source models, LongCat-Flash-Thinking, exhibit competitive performance on complex reasoning benchmarks, matching the capability of most advanced LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_26256 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training Hu, Tianhao Liu, Xiangcheng Xiao, Youshao Zheng, Yang Huang, Xuan Ding, Jinrui Zhang, Yufei Liang, Tao Zang, Hongyu Chen, Quan Sun, Yueqing Shi, Wenjie Zhang, Chao Wang, Wei Gu, Qi Sun, Yerui Xie, Yucheng Cai, Xunliang Machine Learning Distributed, Parallel, and Cluster Computing Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline. Asynchronous training offers a natural remedy by overlapping generation with training, but introduces a fundamental tension between efficiency and algorithmic correctness. We identify three constraints in asynchronous training to preserve convergence: intra-trajectory policy consistency, data integrity, and bounded staleness. Existing approaches fail to intrinsically address the long-tailed trajectory problem, which is further exacerbated by the imbalance characteristic of Mix-of-Experts models, or deviate from the standard RL training formulation, thereby hindering model convergence. Therefore, we propose DORA (Dynamic ORchestration for Asynchronous Rollout), which addresses this challenge through algorithm-system co-design. DORA introduces multi-version streaming rollout, a novel asynchronous paradigm that maintains multiple policy versions concurrently -- simultaneously achieving full bubble elimination without compromising algorithmic constraints. Experimental results demonstrate that our DORA system achieves substantial improvements in throughput -- up to 2--3 times higher than state-of-the-art systems on open-source benchmarks -- without compromising convergence. Furthermore, in large-scale industrial applications with tens of thousands of accelerators, DORA accelerates RL training by 2--4 times compared to synchronous training across various scenarios. The resultant open-source models, LongCat-Flash-Thinking, exhibit competitive performance on complex reasoning benchmarks, matching the capability of most advanced LLMs. |
| title | DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2604.26256 |