Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.11480 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914507014537216 |
|---|---|
| author | Li, Shipeng Yang, Zhiqin Li, Shikun Xia, Xiaobo Liu, Hengyu Zhang, Xinghua Chen, Gaode Fang, Dong Tai, Ying Peng, Zhe |
| author_facet | Li, Shipeng Yang, Zhiqin Li, Shikun Xia, Xiaobo Liu, Hengyu Zhang, Xinghua Chen, Gaode Fang, Dong Tai, Ying Peng, Zhe |
| contents | Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11480 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment Li, Shipeng Yang, Zhiqin Li, Shikun Xia, Xiaobo Liu, Hengyu Zhang, Xinghua Chen, Gaode Fang, Dong Tai, Ying Peng, Zhe Machine Learning Artificial Intelligence Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset. |
| title | LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.11480 |