CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909149872259072 |
|---|---|
| author | Fang, Feiteng Zhu, Liang Yang, Min Feng, Xi Hou, Jinchang Zhao, Qixuan Li, Chengming Hu, Xiping Xu, Ruifeng |
| author_facet | Fang, Feiteng Zhu, Liang Yang, Min Feng, Xi Hou, Jinchang Zhao, Qixuan Li, Chengming Hu, Xiping Xu, Ruifeng |
| contents | Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences, ensuring these LLMs behave in beneficial and comprehensible ways to users. However, a longstanding challenge in human alignment techniques based on reinforcement learning lies in their inherent complexity and difficulty in training. To address this challenge, we present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly. CLHA employs a novel rescoring strategy to evaluate the noise within the data by considering its inherent quality and dynamically adjusting the training process. Simultaneously, CLHA utilizes pairwise contrastive loss and adaptive supervised fine-tuning loss to adaptively modify the likelihood of generating responses, ensuring enhanced alignment with human preferences. Using advanced methods, CLHA surpasses other algorithms, showcasing superior performance in terms of reward model scores, automatic evaluations, and human assessments on the widely used ``Helpful and Harmless'' dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16649 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment Fang, Feiteng Zhu, Liang Yang, Min Feng, Xi Hou, Jinchang Zhao, Qixuan Li, Chengming Hu, Xiping Xu, Ruifeng Artificial Intelligence Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences, ensuring these LLMs behave in beneficial and comprehensible ways to users. However, a longstanding challenge in human alignment techniques based on reinforcement learning lies in their inherent complexity and difficulty in training. To address this challenge, we present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly. CLHA employs a novel rescoring strategy to evaluate the noise within the data by considering its inherent quality and dynamically adjusting the training process. Simultaneously, CLHA utilizes pairwise contrastive loss and adaptive supervised fine-tuning loss to adaptively modify the likelihood of generating responses, ensuring enhanced alignment with human preferences. Using advanced methods, CLHA surpasses other algorithms, showcasing superior performance in terms of reward model scores, automatic evaluations, and human assessments on the widely used ``Helpful and Harmless'' dataset. |
| title | CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2403.16649 |