Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917529640763392 |
|---|---|
| author | Lu, Xiaodong Wang, Xiaohan Chai, Jiajun Yin, Guojun Lin, Wei Chen, Zhijun Luo, Yu Zhuang, Fuzhen Ban, Yikun Wang, Deqing |
| author_facet | Lu, Xiaodong Wang, Xiaohan Chai, Jiajun Yin, Guojun Lin, Wei Chen, Zhijun Luo, Yu Zhuang, Fuzhen Ban, Yikun Wang, Deqing |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models. However, existing RLVR methods utilize rollouts in an indiscriminate and short-horizon manner: responses of heterogeneous quality within each prompt are treated uniformly, and historical rollouts are discarded after a single use. This leads to noisy supervision, poor sample efficiency, and suboptimal policy updates. We address these issues by formulating rollout scheduling in RLVR as a contextual bandit problem and proposing a unified neural scheduling framework that adaptively selects high-value rollouts throughout training. Each rollout is treated as an arm whose reward is defined by the induced performance gain between consecutive optimization steps. The resulting scheduler supports both noise-aware intra-group selection and adaptive global reuse of historical rollouts within a single principled framework. We provide theoretical justification by deriving sublinear regret bounds and showing that enlarging the rollout buffer improves the achievable performance upper bound. Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08499 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards Lu, Xiaodong Wang, Xiaohan Chai, Jiajun Yin, Guojun Lin, Wei Chen, Zhijun Luo, Yu Zhuang, Fuzhen Ban, Yikun Wang, Deqing Machine Learning Artificial Intelligence Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models. However, existing RLVR methods utilize rollouts in an indiscriminate and short-horizon manner: responses of heterogeneous quality within each prompt are treated uniformly, and historical rollouts are discarded after a single use. This leads to noisy supervision, poor sample efficiency, and suboptimal policy updates. We address these issues by formulating rollout scheduling in RLVR as a contextual bandit problem and proposing a unified neural scheduling framework that adaptively selects high-value rollouts throughout training. Each rollout is treated as an arm whose reward is defined by the induced performance gain between consecutive optimization steps. The resulting scheduler supports both noise-aware intra-group selection and adaptive global reuse of historical rollouts within a single principled framework. We provide theoretical justification by deriving sublinear regret bounds and showing that enlarging the rollout buffer improves the achievable performance upper bound. Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods. |
| title | Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.08499 |