Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.02795 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914999171022848 |
|---|---|
| author | Gao, Bofei Song, Feifan Miao, Yibo Cai, Zefan Yang, Zhe Chen, Liang Hu, Helan Xu, Runxin Dong, Qingxiu Zheng, Ce Quan, Shanghaoran Xiao, Wen Zhang, Ge Zan, Daoguang Lu, Keming Yu, Bowen Liu, Dayiheng Cui, Zeyu Yang, Jian Sha, Lei Wang, Houfeng Sui, Zhifang Wang, Peiyi Liu, Tianyu Chang, Baobao |
| author_facet | Gao, Bofei Song, Feifan Miao, Yibo Cai, Zefan Yang, Zhe Chen, Liang Hu, Helan Xu, Runxin Dong, Qingxiu Zheng, Ce Quan, Shanghaoran Xiao, Wen Zhang, Ge Zan, Daoguang Lu, Keming Yu, Bowen Liu, Dayiheng Cui, Zeyu Yang, Jian Sha, Lei Wang, Houfeng Sui, Zhifang Wang, Peiyi Liu, Tianyu Chang, Baobao |
| contents | Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_02795 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards a Unified View of Preference Learning for Large Language Models: A Survey Gao, Bofei Song, Feifan Miao, Yibo Cai, Zefan Yang, Zhe Chen, Liang Hu, Helan Xu, Runxin Dong, Qingxiu Zheng, Ce Quan, Shanghaoran Xiao, Wen Zhang, Ge Zan, Daoguang Lu, Keming Yu, Bowen Liu, Dayiheng Cui, Zeyu Yang, Jian Sha, Lei Wang, Houfeng Sui, Zhifang Wang, Peiyi Liu, Tianyu Chang, Baobao Computation and Language Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences. |
| title | Towards a Unified View of Preference Learning for Large Language Models: A Survey |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.02795 |