The Real, the Better: Aligning Large Language Models with Online Human Behaviors
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909186398355456 |
|---|---|
| author | Jiang, Guanying Yan, Lingyong Shi, Haibo Yin, Dawei |
| author_facet | Jiang, Guanying Yan, Lingyong Shi, Haibo Yin, Dawei |
| contents | Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Real, the Better: Aligning Large Language Models with Online Human Behaviors Jiang, Guanying Yan, Lingyong Shi, Haibo Yin, Dawei Computation and Language Artificial Intelligence Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations. |
| title | The Real, the Better: Aligning Large Language Models with Online Human Behaviors |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.00578 |