Parameter Efficient Reinforcement Learning from Human Feedback
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913498740555776 |
|---|---|
| author | Sidahmed, Hakim Phatale, Samrat Hutcheson, Alex Lin, Zhuonan Chen, Zhang Yu, Zac Jin, Jarvis Chaudhary, Simral Komarytsia, Roman Ahlheim, Christiane Zhu, Yonghao Li, Bowen Ganesh, Saravanan Byrne, Bill Hoffmann, Jessica Mansoor, Hassan Li, Wei Rastogi, Abhinav Dixon, Lucas |
| author_facet | Sidahmed, Hakim Phatale, Samrat Hutcheson, Alex Lin, Zhuonan Chen, Zhang Yu, Zac Jin, Jarvis Chaudhary, Simral Komarytsia, Roman Ahlheim, Christiane Zhu, Yonghao Li, Bowen Ganesh, Saravanan Byrne, Bill Hoffmann, Jessica Mansoor, Hassan Li, Wei Rastogi, Abhinav Dixon, Lucas |
| contents | While Reinforcement Learning from Human Feedback (RLHF) effectively aligns pretrained Large Language and Vision-Language Models (LLMs, and VLMs) with human preferences, its computational cost and complexity hamper its wider adoption. To alleviate some of the computational burden of fine-tuning, parameter efficient methods, like LoRA were introduced. In this work, we empirically evaluate the setup of Parameter Efficient Reinforcement Learning from Human Feedback (PE-RLHF) that leverages LoRA fine-tuning for Reward Modeling, and Reinforcement Learning. We benchmark the PE-RLHF setup on six diverse datasets spanning summarization, harmless/helpful response generation, UI automation, and visual question answering in terms of effectiveness of the trained models, and the training resources required. Our findings show, for the first time, that PE-RLHF achieves comparable performance to RLHF, while significantly reducing training time (up to 90% faster for reward models, and 30% faster for RL), and memory footprint (up to 50% reduction for reward models, and 27% for RL). We provide comprehensive ablations across LoRA ranks, and model sizes for both reward modeling and reinforcement learning. By mitigating the computational burden associated with RLHF, we push for a broader adoption of PE-RLHF as an alignment technique for LLMs and VLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_10704 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Parameter Efficient Reinforcement Learning from Human Feedback Sidahmed, Hakim Phatale, Samrat Hutcheson, Alex Lin, Zhuonan Chen, Zhang Yu, Zac Jin, Jarvis Chaudhary, Simral Komarytsia, Roman Ahlheim, Christiane Zhu, Yonghao Li, Bowen Ganesh, Saravanan Byrne, Bill Hoffmann, Jessica Mansoor, Hassan Li, Wei Rastogi, Abhinav Dixon, Lucas Machine Learning Artificial Intelligence Computation and Language While Reinforcement Learning from Human Feedback (RLHF) effectively aligns pretrained Large Language and Vision-Language Models (LLMs, and VLMs) with human preferences, its computational cost and complexity hamper its wider adoption. To alleviate some of the computational burden of fine-tuning, parameter efficient methods, like LoRA were introduced. In this work, we empirically evaluate the setup of Parameter Efficient Reinforcement Learning from Human Feedback (PE-RLHF) that leverages LoRA fine-tuning for Reward Modeling, and Reinforcement Learning. We benchmark the PE-RLHF setup on six diverse datasets spanning summarization, harmless/helpful response generation, UI automation, and visual question answering in terms of effectiveness of the trained models, and the training resources required. Our findings show, for the first time, that PE-RLHF achieves comparable performance to RLHF, while significantly reducing training time (up to 90% faster for reward models, and 30% faster for RL), and memory footprint (up to 50% reduction for reward models, and 27% for RL). We provide comprehensive ablations across LoRA ranks, and model sizes for both reward modeling and reinforcement learning. By mitigating the computational burden associated with RLHF, we push for a broader adoption of PE-RLHF as an alignment technique for LLMs and VLMs. |
| title | Parameter Efficient Reinforcement Learning from Human Feedback |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2403.10704 |