Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.02554 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911783485177856 |
|---|---|
| author | Cai, Tianchi Song, Xierui Jiang, Jiyan Teng, Fei Gu, Jinjie Zhang, Guannan |
| author_facet | Cai, Tianchi Song, Xierui Jiang, Jiyan Teng, Fei Gu, Jinjie Zhang, Guannan |
| contents | Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_02554 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference Cai, Tianchi Song, Xierui Jiang, Jiyan Teng, Fei Gu, Jinjie Zhang, Guannan Machine Learning Computation and Language Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released. |
| title | ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2312.02554 |