The Art of Scaling Reinforcement Learning Compute for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914095350939648 |
|---|---|
| author | Khatri, Devvrit Madaan, Lovish Tiwari, Rishabh Bansal, Rachit Duvvuri, Sai Surya Zaheer, Manzil Dhillon, Inderjit S. Brandfonbrener, David Agarwal, Rishabh |
| author_facet | Khatri, Devvrit Madaan, Lovish Tiwari, Rishabh Bansal, Rachit Duvvuri, Sai Surya Zaheer, Manzil Dhillon, Inderjit S. Brandfonbrener, David Agarwal, Rishabh |
| contents | Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13786 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Art of Scaling Reinforcement Learning Compute for LLMs Khatri, Devvrit Madaan, Lovish Tiwari, Rishabh Bansal, Rachit Duvvuri, Sai Surya Zaheer, Manzil Dhillon, Inderjit S. Brandfonbrener, David Agarwal, Rishabh Machine Learning Artificial Intelligence Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training. |
| title | The Art of Scaling Reinforcement Learning Compute for LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.13786 |