Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915799775576064 |
|---|---|
| author | Zhang, Zhi Han, Zhen Mavromatis, Costas Zhu, Qi Zhang, Yunyi Guan, Sheng Wang, Dingmin Zhou, Xiong Wang, Shuai Adeshina, Soji Ioannidis, Vassilis Rangwala, Huzefa |
| author_facet | Zhang, Zhi Han, Zhen Mavromatis, Costas Zhu, Qi Zhang, Yunyi Guan, Sheng Wang, Dingmin Zhou, Xiong Wang, Shuai Adeshina, Soji Ioannidis, Vassilis Rangwala, Huzefa |
| contents | Reinforcement learning (RL) plays a central role in large language model (LLM) post-training. Among existing approaches, Group Relative Policy Optimization (GRPO) is widely used, especially for RL with verifiable rewards (RLVR) fine-tuning. In GRPO, each query prompts the LLM to generate a group of rollouts with a fixed group size $N$. When all rollouts in a group share the same outcome, either all correct or all incorrect, the group-normalized advantages become zero, yielding no gradient signal and wasting fine-tuning compute. We introduce Adaptive Efficient Rollout Optimization (AERO), an enhancement of GRPO. AERO uses an adaptive rollout strategy, applies selective rejection to strategically prune rollouts, and maintains a Bayesian posterior to prevent zero-advantage dead zones. Across three model configurations (Qwen2.5-Math-1.5B, Qwen2.5-7B, and Qwen2.5-7B-Instruct), AERO improves compute efficiency without sacrificing performance. Under the same total rollout budget, AERO reduces total training compute by about 48% while shortening wall-clock time per step by about 45% on average. Despite the substantial reduction in compute, AERO matches or improves Pass@8 and Avg@8 over GRPO, demonstrating a practical, scalable, and compute-efficient strategy for RL-based LLM alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14338 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning Zhang, Zhi Han, Zhen Mavromatis, Costas Zhu, Qi Zhang, Yunyi Guan, Sheng Wang, Dingmin Zhou, Xiong Wang, Shuai Adeshina, Soji Ioannidis, Vassilis Rangwala, Huzefa Machine Learning Artificial Intelligence Reinforcement learning (RL) plays a central role in large language model (LLM) post-training. Among existing approaches, Group Relative Policy Optimization (GRPO) is widely used, especially for RL with verifiable rewards (RLVR) fine-tuning. In GRPO, each query prompts the LLM to generate a group of rollouts with a fixed group size $N$. When all rollouts in a group share the same outcome, either all correct or all incorrect, the group-normalized advantages become zero, yielding no gradient signal and wasting fine-tuning compute. We introduce Adaptive Efficient Rollout Optimization (AERO), an enhancement of GRPO. AERO uses an adaptive rollout strategy, applies selective rejection to strategically prune rollouts, and maintains a Bayesian posterior to prevent zero-advantage dead zones. Across three model configurations (Qwen2.5-Math-1.5B, Qwen2.5-7B, and Qwen2.5-7B-Instruct), AERO improves compute efficiency without sacrificing performance. Under the same total rollout budget, AERO reduces total training compute by about 48% while shortening wall-clock time per step by about 45% on average. Despite the substantial reduction in compute, AERO matches or improves Pass@8 and Avg@8 over GRPO, demonstrating a practical, scalable, and compute-efficient strategy for RL-based LLM alignment. |
| title | Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.14338 |