Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917353807151104 |
|---|---|
| author | Nimmaturi, Datta Bhargava, Vaishnavi Ghosh, Rajat George, Johnu Dutta, Debojyoti |
| author_facet | Nimmaturi, Datta Bhargava, Vaishnavi Ghosh, Rajat George, Johnu Dutta, Debojyoti |
| contents | Fine-tuning large language models (LLMs) for reasoning tasks using reinforcement learning methods like Group Relative Policy Optimization (GRPO) is computationally expensive. To address this, we propose a predictive framework that models training dynamics and helps optimize resource usage. Through experiments on Llama and Qwen models (3B 8B), we derive an empirical scaling law based on model size, initial performance, and training progress. This law predicts reward trajectories and identifies three consistent training phases: slow start, rapid improvement, and plateau. We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance. Our approach generalizes across model types, providing a practical guide for efficient GRPO-based fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_18014 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models Nimmaturi, Datta Bhargava, Vaishnavi Ghosh, Rajat George, Johnu Dutta, Debojyoti Machine Learning Fine-tuning large language models (LLMs) for reasoning tasks using reinforcement learning methods like Group Relative Policy Optimization (GRPO) is computationally expensive. To address this, we propose a predictive framework that models training dynamics and helps optimize resource usage. Through experiments on Llama and Qwen models (3B 8B), we derive an empirical scaling law based on model size, initial performance, and training progress. This law predicts reward trajectories and identifies three consistent training phases: slow start, rapid improvement, and plateau. We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance. Our approach generalizes across model types, providing a practical guide for efficient GRPO-based fine-tuning. |
| title | Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.18014 |