RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908532343832576 |
|---|---|
| author | Li, Jiahui Li, Lin Chang, Tai-wei Kuang, Kun Chen, Long Zhou, Jun Yang, Cheng |
| author_facet | Li, Jiahui Li, Lin Chang, Tai-wei Kuang, Kun Chen, Long Zhou, Jun Yang, Cheng |
| contents | Reinforcement learning from human feedback (RLHF) offers a promising approach to aligning large language models (LLMs) with human preferences. Typically, a reward model is trained or supplied to act as a proxy for humans in evaluating generated responses during the reinforcement training phase. However, current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence. This approach may overlook the significant contributions of individual tokens toward the desired outcome. To this end, we propose a more fine-grained, token-level guidance approach for RL training. Specifically, we introduce RED, a novel reward redistribition method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. Utilizing these fine-grained rewards enhances the model's understanding of language nuances, leading to more precise performance improvements. Notably, our method does not require modifying the reward model or introducing additional training steps, thereby incurring minimal computational costs. Experimental results across diverse datasets and tasks demonstrate the superiority of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_08302 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution Li, Jiahui Li, Lin Chang, Tai-wei Kuang, Kun Chen, Long Zhou, Jun Yang, Cheng Computation and Language Artificial Intelligence Reinforcement learning from human feedback (RLHF) offers a promising approach to aligning large language models (LLMs) with human preferences. Typically, a reward model is trained or supplied to act as a proxy for humans in evaluating generated responses during the reinforcement training phase. However, current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence. This approach may overlook the significant contributions of individual tokens toward the desired outcome. To this end, we propose a more fine-grained, token-level guidance approach for RL training. Specifically, we introduce RED, a novel reward redistribition method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. Utilizing these fine-grained rewards enhances the model's understanding of language nuances, leading to more precise performance improvements. Notably, our method does not require modifying the reward model or introducing additional training steps, thereby incurring minimal computational costs. Experimental results across diverse datasets and tasks demonstrate the superiority of our approach. |
| title | RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.08302 |