Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914104633982976 |
|---|---|
| author | Mroueh, Youssef |
| author_facet | Mroueh, Youssef |
| contents | Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $π_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $π_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $π_{n-1}$, as well as the policies $π_{n-1}$ and $π_{\mathrm{ref}}$. Iterating results in a sequence $\{π_n\}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06639 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification Mroueh, Youssef Machine Learning Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $π_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $π_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $π_{n-1}$, as well as the policies $π_{n-1}$ and $π_{\mathrm{ref}}$. Iterating results in a sequence $\{π_n\}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success. |
| title | Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.06639 |