Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929718506291200 |
|---|---|
| author | Jiang, Yuxin Huang, Bo Wang, Yufei Zeng, Xingshan Li, Liangyou Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei |
| author_facet | Jiang, Yuxin Huang, Bo Wang, Yufei Zeng, Xingshan Li, Liangyou Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei |
| contents | Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the generation of the winning response and the losing response within pairwise data are typically isolated, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants. We release our repository at https://github.com/YJiangcm/BMC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07471 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization Jiang, Yuxin Huang, Bo Wang, Yufei Zeng, Xingshan Li, Liangyou Wang, Yasheng Jiang, Xin Shang, Lifeng Tang, Ruiming Wang, Wei Computation and Language Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the generation of the winning response and the losing response within pairwise data are typically isolated, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants. We release our repository at https://github.com/YJiangcm/BMC. |
| title | Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.07471 |