SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915253037563904 |
|---|---|
| author | Zhang, Xiaojiang Wang, Jinghui Cheng, Zifei Zhuang, Wenhao Lin, Zheng Zhang, Minglei Wang, Shaojie Cui, Yinghan Wang, Chao Peng, Junyi Jiang, Shimiao Kuang, Shiqi Yin, Shouyu Wen, Chaohang Zhang, Haotian Chen, Bin Yu, Bing |
| author_facet | Zhang, Xiaojiang Wang, Jinghui Cheng, Zifei Zhuang, Wenhao Lin, Zheng Zhang, Minglei Wang, Shaojie Cui, Yinghan Wang, Chao Peng, Junyi Jiang, Shimiao Kuang, Shiqi Yin, Shouyu Wen, Chaohang Zhang, Haotian Chen, Bin Yu, Bing |
| contents | Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14286 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM Zhang, Xiaojiang Wang, Jinghui Cheng, Zifei Zhuang, Wenhao Lin, Zheng Zhang, Minglei Wang, Shaojie Cui, Yinghan Wang, Chao Peng, Junyi Jiang, Shimiao Kuang, Shiqi Yin, Shouyu Wen, Chaohang Zhang, Haotian Chen, Bin Yu, Bing Machine Learning Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks. |
| title | SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.14286 |