Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.17850 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910003948945408 |
|---|---|
| author | Zhang, Han Zheng, Ruibin Yi, Zexuan Zhang, Zhuo Peng, Hanyang Wang, Hui Yuan, Zike Ke, Cai Chen, Shiwei Yang, Jiacheng Li, Yangning Li, Xiang Yan, Jiangyue Liu, Yaoqi Jing, Liwen Qi, Jiayin Xu, Ruifeng Fang, Binxing Yu, Yue |
| author_facet | Zhang, Han Zheng, Ruibin Yi, Zexuan Zhang, Zhuo Peng, Hanyang Wang, Hui Yuan, Zike Ke, Cai Chen, Shiwei Yang, Jiacheng Li, Yangning Li, Xiang Yan, Jiangyue Liu, Yaoqi Jing, Liwen Qi, Jiayin Xu, Ruifeng Fang, Binxing Yu, Yue |
| contents | As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees. Experiments show GEPO achieves superior stability - only a 3% performance drop from online to 1800s latency-and reduces the best-to-last gap by 85% versus GSPO (1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_17850 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning Zhang, Han Zheng, Ruibin Yi, Zexuan Zhang, Zhuo Peng, Hanyang Wang, Hui Yuan, Zike Ke, Cai Chen, Shiwei Yang, Jiacheng Li, Yangning Li, Xiang Yan, Jiangyue Liu, Yaoqi Jing, Liwen Qi, Jiayin Xu, Ruifeng Fang, Binxing Yu, Yue Machine Learning Artificial Intelligence As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees. Experiments show GEPO achieves superior stability - only a 3% performance drop from online to 1800s latency-and reduces the best-to-last gap by 85% versus GSPO (1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments. |
| title | GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.17850 |