Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912403304742912 |
|---|---|
| author | Mroueh, Youssef Dupuis, Nicolas Belgodere, Brian Nitsure, Apoorva Rigotti, Mattia Greenewald, Kristjan Navratil, Jiri Ross, Jerret Rios, Jesus |
| author_facet | Mroueh, Youssef Dupuis, Nicolas Belgodere, Brian Nitsure, Apoorva Rigotti, Mattia Greenewald, Kristjan Navratil, Jiri Ross, Jerret Rios, Jesus |
| contents | We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22257 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training Mroueh, Youssef Dupuis, Nicolas Belgodere, Brian Nitsure, Apoorva Rigotti, Mattia Greenewald, Kristjan Navratil, Jiri Ross, Jerret Rios, Jesus Machine Learning We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart. |
| title | Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.22257 |