It Takes Two: Your GRPO Is Secretly DPO
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911682477948928 |
|---|---|
| author | Wu, Yihong Ma, Liheng Ding, Lei Li, Muzhi Wang, Xinyu Chen, Kejia Su, Zhan Zhang, Zhanguang Huang, Chenyang Zhang, Yingxue Coates, Mark Nie, Jian-Yun |
| author_facet | Wu, Yihong Ma, Liheng Ding, Lei Li, Muzhi Wang, Xinyu Chen, Kejia Su, Zhan Zhang, Zhanguang Huang, Chenyang Zhang, Yingxue Coates, Mark Nie, Jian-Yun |
| contents | GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the \emph{value baselines} from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains $97.6\%$ of the performance of 16-GRPO, while requiring only $12.5\%$ of the rollouts and $21\%$ of the training time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00977 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | It Takes Two: Your GRPO Is Secretly DPO Wu, Yihong Ma, Liheng Ding, Lei Li, Muzhi Wang, Xinyu Chen, Kejia Su, Zhan Zhang, Zhanguang Huang, Chenyang Zhang, Yingxue Coates, Mark Nie, Jian-Yun Machine Learning Computation and Language GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the \emph{value baselines} from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains $97.6\%$ of the performance of 16-GRPO, while requiring only $12.5\%$ of the rollouts and $21\%$ of the training time. |
| title | It Takes Two: Your GRPO Is Secretly DPO |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.00977 |