APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918149093326848 |
|---|---|
| author | Zhou, Yuzhen Li, Jiajun Su, Yusheng Ramesh, Gowtham Zhu, Zilin Long, Xiang Zhao, Chenyang Pan, Jin Yu, Xiaodong Wang, Ze Du, Kangrui Wu, Jialian Sun, Ximeng Liu, Jiang Yu, Qiaolin Chen, Hao Liu, Zicheng Barsoum, Emad |
| author_facet | Zhou, Yuzhen Li, Jiajun Su, Yusheng Ramesh, Gowtham Zhu, Zilin Long, Xiang Zhao, Chenyang Pan, Jin Yu, Xiaodong Wang, Ze Du, Kangrui Wu, Jialian Sun, Ximeng Liu, Jiang Yu, Qiaolin Chen, Hao Liu, Zicheng Barsoum, Emad |
| contents | Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained language models (LLMs). Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by 22.5% on average (at most 44%) across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves 2.1% on average(at most 8%) higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems. Our codebase is available at https://github.com/RLsys-Foundation/APRIL |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_18521 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation Zhou, Yuzhen Li, Jiajun Su, Yusheng Ramesh, Gowtham Zhu, Zilin Long, Xiang Zhao, Chenyang Pan, Jin Yu, Xiaodong Wang, Ze Du, Kangrui Wu, Jialian Sun, Ximeng Liu, Jiang Yu, Qiaolin Chen, Hao Liu, Zicheng Barsoum, Emad Machine Learning Artificial Intelligence Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained language models (LLMs). Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by 22.5% on average (at most 44%) across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves 2.1% on average(at most 8%) higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems. Our codebase is available at https://github.com/RLsys-Foundation/APRIL |
| title | APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.18521 |