Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918059434835968 |
|---|---|
| author | Cho, Jay Hyeon Oh, JunHyeok Kim, Myunsoo Lee, Byung-Jun |
| author_facet | Cho, Jay Hyeon Oh, JunHyeok Kim, Myunsoo Lee, Byung-Jun |
| contents | Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12725 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking DPO: The Role of Rejected Responses in Preference Misalignment Cho, Jay Hyeon Oh, JunHyeok Kim, Myunsoo Lee, Byung-Jun Artificial Intelligence Computation and Language Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms. |
| title | Rethinking DPO: The Role of Rejected Responses in Preference Misalignment |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.12725 |