LRHP: Learning Representations for Human Preferences via Preference Pairs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917795871064064 |
|---|---|
| author | Wang, Chenglong Gan, Yang Huo, Yifu Mu, Yongyu He, Qiaozhi Yang, Murun Xiao, Tong Zhang, Chunliang Liu, Tongran Zhu, Jingbo |
| author_facet | Wang, Chenglong Gan, Yang Huo, Yifu Mu, Yongyu He, Qiaozhi Yang, Murun Xiao, Tong Zhang, Chunliang Liu, Tongran Zhu, Jingbo |
| contents | To improve human-preference alignment training, current research has developed numerous preference datasets consisting of preference pairs labeled as "preferred" or "dispreferred". These preference pairs are typically used to encode human preferences into a single numerical value through reward modeling, which acts as a reward signal during reinforcement learning from human feedback (RLHF). However, representing these human preferences as a numerical value complicates the analysis of these preferences and restricts their broader applications other than RLHF. In contrast, in this work, we introduce a preference representation learning task that aims to construct a richer and more structured representation of human preferences. We further develop a more generalizable framework, Learning Representations for Human Preferences via preference pairs (namely LRHP), which extends beyond traditional reward modeling to tackle this task. We verify the utility of preference representations in two downstream tasks: preference data selection and preference margin prediction. Building upon the human preferences in representations, we achieve strong performance in both tasks, significantly outperforming baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04503 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LRHP: Learning Representations for Human Preferences via Preference Pairs Wang, Chenglong Gan, Yang Huo, Yifu Mu, Yongyu He, Qiaozhi Yang, Murun Xiao, Tong Zhang, Chunliang Liu, Tongran Zhu, Jingbo Computation and Language Artificial Intelligence To improve human-preference alignment training, current research has developed numerous preference datasets consisting of preference pairs labeled as "preferred" or "dispreferred". These preference pairs are typically used to encode human preferences into a single numerical value through reward modeling, which acts as a reward signal during reinforcement learning from human feedback (RLHF). However, representing these human preferences as a numerical value complicates the analysis of these preferences and restricts their broader applications other than RLHF. In contrast, in this work, we introduce a preference representation learning task that aims to construct a richer and more structured representation of human preferences. We further develop a more generalizable framework, Learning Representations for Human Preferences via preference pairs (namely LRHP), which extends beyond traditional reward modeling to tackle this task. We verify the utility of preference representations in two downstream tasks: preference data selection and preference margin prediction. Building upon the human preferences in representations, we achieve strong performance in both tasks, significantly outperforming baselines. |
| title | LRHP: Learning Representations for Human Preferences via Preference Pairs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.04503 |