Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.19763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913370258538496 |
|---|---|
| author | Liao, Kuo Li, Shuang Zhao, Meng Liu, Liqun Xue, Mengge Hu, Zhenyu Han, Honglin Yin, Chengguo |
| author_facet | Liao, Kuo Li, Shuang Zhao, Meng Liu, Liqun Xue, Mengge Hu, Zhenyu Han, Honglin Yin, Chengguo |
| contents | Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks. To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding. Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks. Code and data available at: https://github.com/MagiaSN/ACL2024_RLLR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19763 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding Liao, Kuo Li, Shuang Zhao, Meng Liu, Liqun Xue, Mengge Hu, Zhenyu Han, Honglin Yin, Chengguo Computation and Language Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks. To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding. Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks. Code and data available at: https://github.com/MagiaSN/ACL2024_RLLR. |
| title | Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.19763 |