Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918452590018560 |
|---|---|
| author | Tan, Zelin Geng, Hejia Yu, Xiaohang Zhang, Mulei Wan, Guancheng Zhou, Yifan He, Qiang Xue, Xiangyuan Zhou, Heng Fan, Yutao Li, Zhongzhi Zhang, Zaibin Zhang, Guibin Zhang, Chen Yin, Zhenfei Torr, Philip Bai, Lei |
| author_facet | Tan, Zelin Geng, Hejia Yu, Xiaohang Zhang, Mulei Wan, Guancheng Zhou, Yifan He, Qiang Xue, Xiangyuan Zhou, Heng Fan, Yutao Li, Zhongzhi Zhang, Zaibin Zhang, Guibin Zhang, Chen Yin, Zhenfei Torr, Philip Bai, Lei |
| contents | While scaling laws for large language models (LLMs) during pre-training have been extensively studied, their behavior under reinforcement learning (RL) post-training remains largely unexplored. This paper presents a systematic empirical investigation of scaling behaviors in RL-based post-training, with a particular focus on mathematical reasoning. Based on a set of experiments across the full Qwen2.5 dense model series (0.5B to 72B), we characterize how model scale, data volume, and computational budget interact to shape performance. Our analysis leads to four key findings: 1. Larger models consistently exhibit superior learning efficiency on both compute and data metrics. 2. The relationship between test loss, compute, and data can be modeled by a predictive power-law which is robust across both base and instruction-tuned models. 3. Although larger models exhibit higher learning efficiency, the analytical learning efficiency term k(N) in the power-law reveals a latent saturation trend in learning efficiency as model size continues to increase. 4. In data-constrained regimes, repeated reuse of high-quality data proves highly effective, as final performance is primarily governed by the total number of optimization steps rather than the uniqueness of samples. Collectively, these results provide a principled foundation and practical guidelines for efficiently scaling the reasoning capabilities of LLMs through RL post-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_25300 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning Tan, Zelin Geng, Hejia Yu, Xiaohang Zhang, Mulei Wan, Guancheng Zhou, Yifan He, Qiang Xue, Xiangyuan Zhou, Heng Fan, Yutao Li, Zhongzhi Zhang, Zaibin Zhang, Guibin Zhang, Chen Yin, Zhenfei Torr, Philip Bai, Lei Machine Learning Artificial Intelligence While scaling laws for large language models (LLMs) during pre-training have been extensively studied, their behavior under reinforcement learning (RL) post-training remains largely unexplored. This paper presents a systematic empirical investigation of scaling behaviors in RL-based post-training, with a particular focus on mathematical reasoning. Based on a set of experiments across the full Qwen2.5 dense model series (0.5B to 72B), we characterize how model scale, data volume, and computational budget interact to shape performance. Our analysis leads to four key findings: 1. Larger models consistently exhibit superior learning efficiency on both compute and data metrics. 2. The relationship between test loss, compute, and data can be modeled by a predictive power-law which is robust across both base and instruction-tuned models. 3. Although larger models exhibit higher learning efficiency, the analytical learning efficiency term k(N) in the power-law reveals a latent saturation trend in learning efficiency as model size continues to increase. 4. In data-constrained regimes, repeated reuse of high-quality data proves highly effective, as final performance is primarily governed by the total number of optimization steps rather than the uniqueness of samples. Collectively, these results provide a principled foundation and practical guidelines for efficiently scaling the reasoning capabilities of LLMs through RL post-training. |
| title | Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.25300 |