DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915772054372352 |
|---|---|
| author | Zhou, Wenxuan Zhang, Shujian Magdalou, Brice Lambert, John Amid, Ehsan Nock, Richard Hard, Andrew |
| author_facet | Zhou, Wenxuan Zhang, Shujian Magdalou, Brice Lambert, John Amid, Ehsan Nock, Richard Hard, Andrew |
| contents | Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking human choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07855 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory Zhou, Wenxuan Zhang, Shujian Magdalou, Brice Lambert, John Amid, Ehsan Nock, Richard Hard, Andrew Machine Learning Artificial Intelligence Computation and Language I.2.6; I.2.7 Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an explicit link with a specific normative model of human choice. Our paper elevates this connection to the full generality of DPO's normative framework. Getting there requires reworking human choice theory's textbook path for a better RLHF/ML fit. It elevates the connection to a remarkably broad viewpoint on preference optimization, considering the current panorama of DPO follow-ups. It also unveils unexpected riches for ML, chief among which the support for non-convex losses, the fact that any compliant ML analytical choice can be embedded with any human choice model, and a normative framework's umbrella wide enough to safeguard DPO's extensions (margins, length correction, ...). A toy experiment ``far away'' from the DPO crowd is given. |
| title | DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2507.07855 |