Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912410849247232 |
|---|---|
| author | Zheng, Haizhong Zhou, Yang Bartoldson, Brian R. Kailkhura, Bhavya Lai, Fan Zhao, Jiawei Chen, Beidi |
| author_facet | Zheng, Haizhong Zhou, Yang Bartoldson, Brian R. Kailkhura, Bhavya Lai, Fan Zhao, Jiawei Chen, Beidi |
| contents | Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02177 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts Zheng, Haizhong Zhou, Yang Bartoldson, Brian R. Kailkhura, Bhavya Lai, Fan Zhao, Jiawei Chen, Beidi Artificial Intelligence Machine Learning Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation. |
| title | Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.02177 |