RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911258111901696 |
|---|---|
| author | Zeng, Zhiyuan Ivison, Hamish Wang, Yiping Yuan, Lifan Li, Shuyue Stella Ye, Zhuorui Li, Siting He, Jacqueline Zhou, Runlong Chen, Tong Zhao, Chenyang Tsvetkov, Yulia Du, Simon Shaolei Jaques, Natasha Peng, Hao Koh, Pang Wei Hajishirzi, Hannaneh |
| author_facet | Zeng, Zhiyuan Ivison, Hamish Wang, Yiping Yuan, Lifan Li, Shuyue Stella Ye, Zhuorui Li, Siting He, Jacqueline Zhou, Runlong Chen, Tong Zhao, Chenyang Tsvetkov, Yulia Du, Simon Shaolei Jaques, Natasha Peng, Hao Koh, Pang Wei Hajishirzi, Hannaneh |
| contents | We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07317 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments Zeng, Zhiyuan Ivison, Hamish Wang, Yiping Yuan, Lifan Li, Shuyue Stella Ye, Zhuorui Li, Siting He, Jacqueline Zhou, Runlong Chen, Tong Zhao, Chenyang Tsvetkov, Yulia Du, Simon Shaolei Jaques, Natasha Peng, Hao Koh, Pang Wei Hajishirzi, Hannaneh Computation and Language Machine Learning We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly. |
| title | RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2511.07317 |