Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2408.15339 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914540879347712 |
|---|---|
| author | Wang, Zhichao Bi, Bin Huang, Can Pentyala, Shiva Kumar Zhu, Zixu James Asur, Sitaram Cheng, Na Claire Wan, Cheng Nie, Dong Hong, Lingzi |
| author_facet | Wang, Zhichao Bi, Bin Huang, Can Pentyala, Shiva Kumar Zhu, Zixu James Asur, Sitaram Cheng, Na Claire Wan, Cheng Nie, Dong Hong, Lingzi |
| contents | RL alignment methods, including RLHF and DPO, are primarily based on pairwise preference data. Although scalar or score-based feedback has been collected in some settings, it is rarely used directly, and preference magnitude information is typically ignored. Furthermore, current alignment frameworks offer limited capability for unifying heterogeneous supervision signals, making it difficult to jointly leverage diverse data types within a single training paradigm. This limitation constrains the richness and scalability of the alignment process. To address this gap, we propose a \textbf{UN}ified \textbf{A}lignment (UNA) framework capable of training across different types of feedback, including binary, pairwise, and score-based, through a generalized implicit reward function. The reward function is theoretically proved to be the optimal policy by the log sum inequality. Extensive experiments on classical benchmarks consistently demonstrate the advantage of the proposed unified framework with typical LLM base models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15339 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Types Wang, Zhichao Bi, Bin Huang, Can Pentyala, Shiva Kumar Zhu, Zixu James Asur, Sitaram Cheng, Na Claire Wan, Cheng Nie, Dong Hong, Lingzi Machine Learning Computation and Language RL alignment methods, including RLHF and DPO, are primarily based on pairwise preference data. Although scalar or score-based feedback has been collected in some settings, it is rarely used directly, and preference magnitude information is typically ignored. Furthermore, current alignment frameworks offer limited capability for unifying heterogeneous supervision signals, making it difficult to jointly leverage diverse data types within a single training paradigm. This limitation constrains the richness and scalability of the alignment process. To address this gap, we propose a \textbf{UN}ified \textbf{A}lignment (UNA) framework capable of training across different types of feedback, including binary, pairwise, and score-based, through a generalized implicit reward function. The reward function is theoretically proved to be the optimal policy by the log sum inequality. Extensive experiments on classical benchmarks consistently demonstrate the advantage of the proposed unified framework with typical LLM base models. |
| title | UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Types |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2408.15339 |