RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915763410960384 |
|---|---|
| author | Yi, Hongzhu Wang, Xinming zhang, Zhenghao Zong, Tianyu Wang, Yuanxiang Xie, Jun Yu, Tao Jin, Haopeng Xu, Kaixin Chen, Feng Chen, Jiahuan Yang, Yujia Guan, Zhenyu Shi, Bingkang Xu, Jungang |
| author_facet | Yi, Hongzhu Wang, Xinming zhang, Zhenghao Zong, Tianyu Wang, Yuanxiang Xie, Jun Yu, Tao Jin, Haopeng Xu, Kaixin Chen, Feng Chen, Jiahuan Yang, Yujia Guan, Zhenyu Shi, Bingkang Xu, Jungang |
| contents | Within the domain of large language models, reinforcement fine-tuning algorithms necessitate the generation of a complete reasoning trajectory beginning from the input query, which incurs significant computational overhead during the rollout phase of training. To address this issue, we analyze the impact of different segments of the reasoning path on the correctness of the final result and, based on these insights, propose Reinforcement Fine-Tuning with Partial Reasoning Optimization (RPO), a plug-and-play reinforcement fine-tuning algorithm. Unlike traditional reinforcement fine-tuning algorithms that generate full reasoning paths, RPO trains the model by generating suffixes of the reasoning path using experience cache. During the rollout phase of training, RPO reduces token generation in this phase by approximately 95%, greatly lowering the theoretical time overhead. Compared with full-path reinforcement fine-tuning algorithms, RPO reduces the training time of the 1.5B model by 90% and the 7B model by 72%. At the same time, it can be integrated with typical algorithms such as GRPO and DAPO, enabling them to achieve training acceleration while maintaining performance comparable to the original algorithms. Our code is open-sourced at https://github.com/yhz5613813/RPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19404 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization Yi, Hongzhu Wang, Xinming zhang, Zhenghao Zong, Tianyu Wang, Yuanxiang Xie, Jun Yu, Tao Jin, Haopeng Xu, Kaixin Chen, Feng Chen, Jiahuan Yang, Yujia Guan, Zhenyu Shi, Bingkang Xu, Jungang Artificial Intelligence Machine Learning Within the domain of large language models, reinforcement fine-tuning algorithms necessitate the generation of a complete reasoning trajectory beginning from the input query, which incurs significant computational overhead during the rollout phase of training. To address this issue, we analyze the impact of different segments of the reasoning path on the correctness of the final result and, based on these insights, propose Reinforcement Fine-Tuning with Partial Reasoning Optimization (RPO), a plug-and-play reinforcement fine-tuning algorithm. Unlike traditional reinforcement fine-tuning algorithms that generate full reasoning paths, RPO trains the model by generating suffixes of the reasoning path using experience cache. During the rollout phase of training, RPO reduces token generation in this phase by approximately 95%, greatly lowering the theoretical time overhead. Compared with full-path reinforcement fine-tuning algorithms, RPO reduces the training time of the 1.5B model by 90% and the 7B model by 72%. At the same time, it can be integrated with typical algorithms such as GRPO and DAPO, enabling them to achieve training acceleration while maintaining performance comparable to the original algorithms. Our code is open-sourced at https://github.com/yhz5613813/RPO. |
| title | RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.19404 |