Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2504.05118 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912320283738112 |
|---|---|
| author | Yue, Yu Yuan, Yufeng Yu, Qiying Zuo, Xiaochen Zhu, Ruofei Xu, Wenyuan Chen, Jiaze Wang, Chengyi Fan, TianTian Du, Zhengyin Wei, Xiangpeng Yu, Xiangyu Liu, Gaohong Liu, Juncai Liu, Lingjun Lin, Haibin Lin, Zhiqi Ma, Bole Zhang, Chi Zhang, Mofan Zhang, Wang Zhu, Hang Zhang, Ru Liu, Xin Wang, Mingxuan Wu, Yonghui Yan, Lin |
| author_facet | Yue, Yu Yuan, Yufeng Yu, Qiying Zuo, Xiaochen Zhu, Ruofei Xu, Wenyuan Chen, Jiaze Wang, Chengyi Fan, TianTian Du, Zhengyin Wei, Xiangpeng Yu, Xiangyu Liu, Gaohong Liu, Juncai Liu, Lingjun Lin, Haibin Lin, Zhiqi Ma, Bole Zhang, Chi Zhang, Mofan Zhang, Wang Zhu, Hang Zhang, Ru Liu, Xin Wang, Mingxuan Wu, Yonghui Yan, Lin |
| contents | We present VAPO, Value-based Augmented Proximal Policy Optimization framework for reasoning models., a novel framework tailored for reasoning models within the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the Qwen 32B pre-trained model, attains a state-of-the-art score of $\mathbf{60.4}$. In direct comparison under identical experimental settings, VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B and DAPO by more than 10 points. The training process of VAPO stands out for its stability and efficiency. It reaches state-of-the-art performance within a mere 5,000 steps. Moreover, across multiple independent runs, no training crashes occur, underscoring its reliability. This research delves into long chain-of-thought (long-CoT) reasoning using a value-based reinforcement learning framework. We pinpoint three key challenges that plague value-based methods: value model bias, the presence of heterogeneous sequence lengths, and the sparsity of reward signals. Through systematic design, VAPO offers an integrated solution that effectively alleviates these challenges, enabling enhanced performance in long-CoT reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05118 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks Yue, Yu Yuan, Yufeng Yu, Qiying Zuo, Xiaochen Zhu, Ruofei Xu, Wenyuan Chen, Jiaze Wang, Chengyi Fan, TianTian Du, Zhengyin Wei, Xiangpeng Yu, Xiangyu Liu, Gaohong Liu, Juncai Liu, Lingjun Lin, Haibin Lin, Zhiqi Ma, Bole Zhang, Chi Zhang, Mofan Zhang, Wang Zhu, Hang Zhang, Ru Liu, Xin Wang, Mingxuan Wu, Yonghui Yan, Lin Artificial Intelligence We present VAPO, Value-based Augmented Proximal Policy Optimization framework for reasoning models., a novel framework tailored for reasoning models within the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the Qwen 32B pre-trained model, attains a state-of-the-art score of $\mathbf{60.4}$. In direct comparison under identical experimental settings, VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B and DAPO by more than 10 points. The training process of VAPO stands out for its stability and efficiency. It reaches state-of-the-art performance within a mere 5,000 steps. Moreover, across multiple independent runs, no training crashes occur, underscoring its reliability. This research delves into long chain-of-thought (long-CoT) reasoning using a value-based reinforcement learning framework. We pinpoint three key challenges that plague value-based methods: value model bias, the presence of heterogeneous sequence lengths, and the sparsity of reward signals. Through systematic design, VAPO offers an integrated solution that effectively alleviates these challenges, enabling enhanced performance in long-CoT reasoning tasks. |
| title | VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2504.05118 |