Reinforcement Learning on Pre-Training Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918147804626944 |
|---|---|
| author | Li, Siheng Li, Kejiao Xu, Zenan Huang, Guanhua Yang, Evander Li, Kun Wu, Haoyuan Wu, Jiajia Zheng, Zihao Zhang, Chenchen Shi, Kun Deng, Kyrierl Yi, Qi Xiong, Ruibin Xu, Tingqiang Jiang, Yuhao Yan, Jianfeng Zeng, Yuyuan Xu, Guanghui Xue, Jinbao Xu, Zhijiang Fang, Zheng Li, Shuai Liu, Qibin Li, Xiaoxue Li, Zhuoyu Tao, Yangyu Gao, Fei Jiang, Cheng Wang, Bo Chao Liu, Kai Zhu, Jianchen Lam, Wai Wang, Wayyt Zhou, Bo Wang, Di |
| author_facet | Li, Siheng Li, Kejiao Xu, Zenan Huang, Guanhua Yang, Evander Li, Kun Wu, Haoyuan Wu, Jiajia Zheng, Zihao Zhang, Chenchen Shi, Kun Deng, Kyrierl Yi, Qi Xiong, Ruibin Xu, Tingqiang Jiang, Yuhao Yan, Jianfeng Zeng, Yuyuan Xu, Guanghui Xue, Jinbao Xu, Zhijiang Fang, Zheng Li, Shuai Liu, Qibin Li, Xiaoxue Li, Zhuoyu Tao, Yangyu Gao, Fei Jiang, Cheng Wang, Bo Chao Liu, Kai Zhu, Jianchen Lam, Wai Wang, Wayyt Zhou, Bo Wang, Di |
| contents | The growing disparity between the exponential scaling of computational resources and the finite growth of high-quality text data now constrains conventional scaling approaches for large language models (LLMs). To address this challenge, we introduce Reinforcement Learning on Pre-Training data (RLPT), a new training-time scaling paradigm for optimizing LLMs. In contrast to prior approaches that scale training primarily through supervised learning, RLPT enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL). While existing RL strategies such as reinforcement learning from human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR) rely on human annotation for reward construction, RLPT eliminates this dependency by deriving reward signals directly from pre-training data. Specifically, it adopts a next-segment reasoning objective, rewarding the policy for accurately predicting subsequent text segments conditioned on the preceding context. This formulation allows RL to be scaled on pre-training data, encouraging the exploration of richer trajectories across broader contexts and thereby fostering more generalizable reasoning skills. Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of RLPT. For example, when applied to Qwen3-4B-Base, RLPT yields absolute improvements of $3.0$, $5.1$, $8.1$, $6.0$, $6.6$, and $5.3$ on MMLU, MMLU-Pro, GPQA-Diamond, KOR-Bench, AIME24, and AIME25, respectively. The results further demonstrate favorable scaling behavior, suggesting strong potential for continued gains with more compute. In addition, RLPT provides a solid foundation, extending the reasoning boundaries of LLMs and enhancing RLVR performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19249 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning on Pre-Training Data Li, Siheng Li, Kejiao Xu, Zenan Huang, Guanhua Yang, Evander Li, Kun Wu, Haoyuan Wu, Jiajia Zheng, Zihao Zhang, Chenchen Shi, Kun Deng, Kyrierl Yi, Qi Xiong, Ruibin Xu, Tingqiang Jiang, Yuhao Yan, Jianfeng Zeng, Yuyuan Xu, Guanghui Xue, Jinbao Xu, Zhijiang Fang, Zheng Li, Shuai Liu, Qibin Li, Xiaoxue Li, Zhuoyu Tao, Yangyu Gao, Fei Jiang, Cheng Wang, Bo Chao Liu, Kai Zhu, Jianchen Lam, Wai Wang, Wayyt Zhou, Bo Wang, Di Computation and Language Artificial Intelligence Machine Learning The growing disparity between the exponential scaling of computational resources and the finite growth of high-quality text data now constrains conventional scaling approaches for large language models (LLMs). To address this challenge, we introduce Reinforcement Learning on Pre-Training data (RLPT), a new training-time scaling paradigm for optimizing LLMs. In contrast to prior approaches that scale training primarily through supervised learning, RLPT enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL). While existing RL strategies such as reinforcement learning from human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR) rely on human annotation for reward construction, RLPT eliminates this dependency by deriving reward signals directly from pre-training data. Specifically, it adopts a next-segment reasoning objective, rewarding the policy for accurately predicting subsequent text segments conditioned on the preceding context. This formulation allows RL to be scaled on pre-training data, encouraging the exploration of richer trajectories across broader contexts and thereby fostering more generalizable reasoning skills. Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of RLPT. For example, when applied to Qwen3-4B-Base, RLPT yields absolute improvements of $3.0$, $5.1$, $8.1$, $6.0$, $6.6$, and $5.3$ on MMLU, MMLU-Pro, GPQA-Diamond, KOR-Bench, AIME24, and AIME25, respectively. The results further demonstrate favorable scaling behavior, suggesting strong potential for continued gains with more compute. In addition, RLPT provides a solid foundation, extending the reasoning boundaries of LLMs and enhancing RLVR performance. |
| title | Reinforcement Learning on Pre-Training Data |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.19249 |