Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916150076506112 |
|---|---|
| author | Zhu, Yu Sun, Chuxiong Yang, Wenfei Wei, Wenqiang Tang, Bo Zhang, Tianzhu Li, Zhiyu Zhang, Shifeng Xiong, Feiyu Hu, Jie yang, Mingchuan |
| author_facet | Zhu, Yu Sun, Chuxiong Yang, Wenfei Wei, Wenqiang Tang, Bo Zhang, Tianzhu Li, Zhiyu Zhang, Shifeng Xiong, Feiyu Hu, Jie yang, Mingchuan |
| contents | Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04283 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy Zhu, Yu Sun, Chuxiong Yang, Wenfei Wei, Wenqiang Tang, Bo Zhang, Tianzhu Li, Zhiyu Zhang, Shifeng Xiong, Feiyu Hu, Jie yang, Mingchuan Computation and Language Artificial Intelligence Machine Learning Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1\% of the training parameters of other methods. |
| title | Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2403.04283 |