Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2501.11651 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909648036036608 |
|---|---|
| author | Hou, Zhenyu Lv, Xin Lu, Rui Zhang, Jiajie Li, Yujiang Yao, Zijun Li, Juanzi Tang, Jie Dong, Yuxiao |
| author_facet | Hou, Zhenyu Lv, Xin Lu, Rui Zhang, Jiajie Li, Yujiang Yao, Zijun Li, Juanzi Tang, Jie Dong, Yuxiao |
| contents | Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11651 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling Hou, Zhenyu Lv, Xin Lu, Rui Zhang, Jiajie Li, Yujiang Yao, Zijun Li, Juanzi Tang, Jie Dong, Yuxiao Machine Learning Computation and Language Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification. |
| title | T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2501.11651 |