2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912085941682176 |
|---|---|
| author | Li, Shilong He, Yancheng Huang, Hui Bu, Xingyuan Liu, Jiaheng Guo, Hangyu Wang, Weixun Gu, Jihao Su, Wenbo Zheng, Bo |
| author_facet | Li, Shilong He, Yancheng Huang, Hui Bu, Xingyuan Liu, Jiaheng Guo, Hangyu Wang, Weixun Gu, Jihao Su, Wenbo Zheng, Bo |
| contents | Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19720 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision Li, Shilong He, Yancheng Huang, Hui Bu, Xingyuan Liu, Jiaheng Guo, Hangyu Wang, Weixun Gu, Jihao Su, Wenbo Zheng, Bo Computation and Language Artificial Intelligence Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences. |
| title | 2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.19720 |