Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12429 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915620884316160 |
|---|---|
| author | Yao, Yihang Zeng, Guangtao Wu, Raina Zhang, Yang Zhao, Ding Hong, Zhang-Wei Gan, Chuang |
| author_facet | Yao, Yihang Zeng, Guangtao Wu, Raina Zhang, Yang Zhao, Ding Hong, Zhang-Wei Gan, Chuang |
| contents | Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12429 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tailored Primitive Initialization is the Secret Key to Reinforcement Learning Yao, Yihang Zeng, Guangtao Wu, Raina Zhang, Yang Zhao, Ding Hong, Zhang-Wei Gan, Chuang Machine Learning Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance. |
| title | Tailored Primitive Initialization is the Secret Key to Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2511.12429 |