$β$-DPO: Direct Preference Optimization with Dynamic $β$
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912070908248064 |
|---|---|
| author | Wu, Junkang Xie, Yuexiang Yang, Zhengyi Wu, Jiancan Gao, Jinyang Ding, Bolin Wang, Xiang He, Xiangnan |
| author_facet | Wu, Junkang Xie, Yuexiang Yang, Zhengyi Wu, Jiancan Gao, Jinyang Ding, Bolin Wang, Xiang He, Xiangnan |
| contents | Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter $β$, as well as to the quality of the preference data. We analyze the impact of $β$ and data quality on DPO, uncovering that optimal $β$ values vary with the informativeness of pairwise data. Addressing the limitations of static $β$ values, we introduce a novel framework that dynamically calibrates $β$ at the batch level, informed by data quality considerations. Additionally, our method incorporates $β$-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic $β$ adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at \url{https://github.com/junkangwu/beta-DPO}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08639 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $β$-DPO: Direct Preference Optimization with Dynamic $β$ Wu, Junkang Xie, Yuexiang Yang, Zhengyi Wu, Jiancan Gao, Jinyang Ding, Bolin Wang, Xiang He, Xiangnan Artificial Intelligence Machine Learning Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter $β$, as well as to the quality of the preference data. We analyze the impact of $β$ and data quality on DPO, uncovering that optimal $β$ values vary with the informativeness of pairwise data. Addressing the limitations of static $β$ values, we introduce a novel framework that dynamically calibrates $β$ at the batch level, informed by data quality considerations. Additionally, our method incorporates $β$-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic $β$ adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at \url{https://github.com/junkangwu/beta-DPO}. |
| title | $β$-DPO: Direct Preference Optimization with Dynamic $β$ |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2407.08639 |