Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.12854 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913961664839680 |
|---|---|
| author | Tu, Songjun Lin, Jiahao Tian, Xiangyu Zhang, Qichao Li, Linjing Fu, Yuqian Xu, Nan He, Wei Lan, Xiangyuan Jiang, Dongmei Zhao, Dongbin |
| author_facet | Tu, Songjun Lin, Jiahao Tian, Xiangyu Zhang, Qichao Li, Linjing Fu, Yuqian Xu, Nan He, Wei Lan, Xiangyuan Jiang, Dongmei Zhao, Dongbin |
| contents | Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12854 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation Tu, Songjun Lin, Jiahao Tian, Xiangyu Zhang, Qichao Li, Linjing Fu, Yuqian Xu, Nan He, Wei Lan, Xiangyuan Jiang, Dongmei Zhao, Dongbin Computation and Language Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations. |
| title | Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.12854 |