Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915084957122560 |
|---|---|
| author | Zhang, Jianfei Bai, Jun Li, Bei Wang, Yanmeng Li, Rumei Lin, Chenghua Rong, Wenge |
| author_facet | Zhang, Jianfei Bai, Jun Li, Bei Wang, Yanmeng Li, Rumei Lin, Chenghua Rong, Wenge |
| contents | Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20834 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment Zhang, Jianfei Bai, Jun Li, Bei Wang, Yanmeng Li, Rumei Lin, Chenghua Rong, Wenge Computation and Language Artificial Intelligence Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them. |
| title | Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.20834 |