Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xiao, Teng, Yuan, Yige, Zhu, Huaisheng, Li, Mingxiao, Honavar, Vasant G
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866915071103336448
author	Xiao, Teng Yuan, Yige Zhu, Huaisheng Li, Mingxiao Honavar, Vasant G
author_facet	Xiao, Teng Yuan, Yige Zhu, Huaisheng Li, Mingxiao Honavar, Vasant G
contents	We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
format	Preprint
id	arxiv_https___arxiv_org_abs_2412_14516
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment Xiao, Teng Yuan, Yige Zhu, Huaisheng Li, Mingxiao Honavar, Vasant G Machine Learning Computation and Language We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
title	Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
topic	Machine Learning Computation and Language
url	https://arxiv.org/abs/2412.14516