VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912602474414080 |
|---|---|
| author | Jiang, Guochao Feng, Wenfeng Quan, Guofeng Hao, Chuzhan Zhang, Yuewei Liu, Guohua Wang, Hao |
| author_facet | Jiang, Guochao Feng, Wenfeng Quan, Guofeng Hao, Chuzhan Zhang, Yuewei Liu, Guohua Wang, Hao |
| contents | Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19803 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models Jiang, Guochao Feng, Wenfeng Quan, Guofeng Hao, Chuzhan Zhang, Yuewei Liu, Guohua Wang, Hao Machine Learning Computation and Language Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines. |
| title | VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.19803 |