Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08007 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910996600193024 |
|---|---|
| author | Dong, Qingxiu Dong, Li Tang, Yao Ye, Tianzhu Sun, Yutao Sui, Zhifang Wei, Furu |
| author_facet | Dong, Qingxiu Dong, Li Tang, Yao Ye, Tianzhu Sun, Yutao Sui, Zhifang Wei, Furu |
| contents | In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08007 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reinforcement Pre-Training Dong, Qingxiu Dong, Li Tang, Yao Ye, Tianzhu Sun, Yutao Sui, Zhifang Wei, Furu Computation and Language In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training. |
| title | Reinforcement Pre-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.08007 |