One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915762223972352 |
|---|---|
| author | Zhao, Weisong Wang, Tong Tan, Zichang Yang, Te Peng, Siran Zhang, Haoyuan Zhang, Tianshuo Shi, Haichao Meng, Meng Yang, Yang Zhu, Xiangyu Lei, Zhen Zhang, Xiao-Yu Zhou, Xu |
| author_facet | Zhao, Weisong Wang, Tong Tan, Zichang Yang, Te Peng, Siran Zhang, Haoyuan Zhang, Tianshuo Shi, Haichao Meng, Meng Yang, Yang Zhu, Xiangyu Lei, Zhen Zhang, Xiao-Yu Zhou, Xu |
| contents | Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the geometric mean of GMPO. While GMPO improves stability by constraining a conservative objective, it shares a fundamental limitation with GRPO: reliance on a fixed aggregation geometry that ignores the evolving and heterogeneous nature of each trajectory. In this work, we unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry via the power-mean geometry exponent p. Within this framework, GRPO and GMPO are recovered as special cases. Theoretically, we demonstrate that adjusting p modulates the concentration of gradient updates, effectively reweighting tokens based on their advantage contribution. To determine p adaptively, we introduce a Clip-aware Effective Sample Size (ESS) mechanism. Specifically, we propose a deterministic rule that maps a trajectory clipping fraction to a target ESS. Then, we solve for the specific p to align the trajectory induced ESS with this target one. This allows PMPO to dynamically transition between the aggressive arithmetic mean for reliable trajectories and the conservative geometric mean for unstable ones. Experiments on multiple mathematical reasoning benchmarks demonstrate that PMPO outperforms strong baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22521 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry Zhao, Weisong Wang, Tong Tan, Zichang Yang, Te Peng, Siran Zhang, Haoyuan Zhang, Tianshuo Shi, Haichao Meng, Meng Yang, Yang Zhu, Xiangyu Lei, Zhen Zhang, Xiao-Yu Zhou, Xu Computation and Language Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the geometric mean of GMPO. While GMPO improves stability by constraining a conservative objective, it shares a fundamental limitation with GRPO: reliance on a fixed aggregation geometry that ignores the evolving and heterogeneous nature of each trajectory. In this work, we unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry via the power-mean geometry exponent p. Within this framework, GRPO and GMPO are recovered as special cases. Theoretically, we demonstrate that adjusting p modulates the concentration of gradient updates, effectively reweighting tokens based on their advantage contribution. To determine p adaptively, we introduce a Clip-aware Effective Sample Size (ESS) mechanism. Specifically, we propose a deterministic rule that maps a trajectory clipping fraction to a target ESS. Then, we solve for the specific p to align the trajectory induced ESS with this target one. This allows PMPO to dynamically transition between the aggressive arithmetic mean for reliable trajectories and the conservative geometric mean for unstable ones. Experiments on multiple mathematical reasoning benchmarks demonstrate that PMPO outperforms strong baselines. |
| title | One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.22521 |