Automatic Pair Construction for Contrastive Post-training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914738737250304 |
|---|---|
| author | Xu, Canwen Rosset, Corby Chau, Ethan C. Del Corro, Luciano Mahajan, Shweti McAuley, Julian Neville, Jennifer Awadallah, Ahmed Hassan Rao, Nikhil |
| author_facet | Xu, Canwen Rosset, Corby Chau, Ethan C. Del Corro, Luciano Mahajan, Shweti McAuley, Julian Neville, Jennifer Awadallah, Ahmed Hassan Rao, Nikhil |
| contents | Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from "easier" pairs and transitioning to "harder" ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02263 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Automatic Pair Construction for Contrastive Post-training Xu, Canwen Rosset, Corby Chau, Ethan C. Del Corro, Luciano Mahajan, Shweti McAuley, Julian Neville, Jennifer Awadallah, Ahmed Hassan Rao, Nikhil Computation and Language Artificial Intelligence Machine Learning Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from "easier" pairs and transitioning to "harder" ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT. |
| title | Automatic Pair Construction for Contrastive Post-training |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2310.02263 |