Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.18573 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908601131466752 |
|---|---|
| author | Liao, Mengqi Xi, Xiangyu Chen, Ruinian Leng, Jia Hu, Yangen Zeng, Ke Liu, Shuai Wan, Huaiyu |
| author_facet | Liao, Mengqi Xi, Xiangyu Chen, Ruinian Leng, Jia Hu, Yangen Zeng, Ke Liu, Shuai Wan, Huaiyu |
| contents | Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18573 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs Liao, Mengqi Xi, Xiangyu Chen, Ruinian Leng, Jia Hu, Yangen Zeng, Ke Liu, Shuai Wan, Huaiyu Machine Learning Computation and Language Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs |
| title | Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2505.18573 |