Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916847544172544 |
|---|---|
| author | Liu, Mingjie Diao, Shizhe Hu, Jian Lu, Ximing Dong, Xin Zhang, Hao Bukharin, Alexander Zhang, Shaokun Zeng, Jiaqi Sreedhar, Makesh Narsimhan Shen, Gerald Mosallanezhad, David Zhang, Di Yang, Jonas Yang, June Kuchaiev, Oleksii Liu, Guilin Yu, Zhiding Molchanov, Pavlo Choi, Yejin Kautz, Jan Dong, Yi |
| author_facet | Liu, Mingjie Diao, Shizhe Hu, Jian Lu, Ximing Dong, Xin Zhang, Hao Bukharin, Alexander Zhang, Shaokun Zeng, Jiaqi Sreedhar, Makesh Narsimhan Shen, Gerald Mosallanezhad, David Zhang, Di Yang, Jonas Yang, June Kuchaiev, Oleksii Liu, Guilin Yu, Zhiding Molchanov, Pavlo Choi, Yejin Kautz, Jan Dong, Yi |
| contents | Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12507 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training Liu, Mingjie Diao, Shizhe Hu, Jian Lu, Ximing Dong, Xin Zhang, Hao Bukharin, Alexander Zhang, Shaokun Zeng, Jiaqi Sreedhar, Makesh Narsimhan Shen, Gerald Mosallanezhad, David Zhang, Di Yang, Jonas Yang, June Kuchaiev, Oleksii Liu, Guilin Yu, Zhiding Molchanov, Pavlo Choi, Yejin Kautz, Jan Dong, Yi Machine Learning Artificial Intelligence Computation and Language Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly. |
| title | Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2507.12507 |