SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912942877835264 |
|---|---|
| author | Zhang, Ruiqi Arora, Daman Mei, Song Zanette, Andrea |
| author_facet | Zhang, Ruiqi Arora, Daman Mei, Song Zanette, Andrea |
| contents | Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt sampling. We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency. Theoretically, we establish that intermediate-difficulty prompts improve the gradient estimator's signal-to-noise ratio, accelerating convergence. Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09016 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning Zhang, Ruiqi Arora, Daman Mei, Song Zanette, Andrea Machine Learning Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt sampling. We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency. Theoretically, we establish that intermediate-difficulty prompts improve the gradient estimator's signal-to-noise ratio, accelerating convergence. Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms. |
| title | SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.09016 |