Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918005174173696 |
|---|---|
| author | Xu, Haoran Peng, Baolin Awadalla, Hany Chen, Dongdong Chen, Yen-Chun Gao, Mei Kim, Young Jin Li, Yunsheng Ren, Liliang Shen, Yelong Wang, Shuohang Xu, Weijian Gao, Jianfeng Chen, Weizhu |
| author_facet | Xu, Haoran Peng, Baolin Awadalla, Hany Chen, Dongdong Chen, Yen-Chun Gao, Mei Kim, Young Jin Li, Yunsheng Ren, Liliang Shen, Yelong Wang, Shuohang Xu, Weijian Gao, Jianfeng Chen, Weizhu |
| contents | Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs) by training them to explicitly generate intermediate reasoning steps. While LLMs readily benefit from such techniques, improving reasoning in Small Language Models (SLMs) remains challenging due to their limited model capacity. Recent work by Deepseek-R1 demonstrates that distillation from LLM-generated synthetic data can substantially improve the reasoning ability of SLM. However, the detailed modeling recipe is not disclosed. In this work, we present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward. We apply our method on Phi-4-Mini, a compact 3.8B-parameter model. The resulting Phi-4-Mini-Reasoning model exceeds, on math reasoning tasks, much larger reasoning models, e.g., outperforming DeepSeek-R1-Distill-Qwen-7B by 3.2 points and DeepSeek-R1-Distill-Llama-8B by 7.7 points on Math-500. Our results validate that a carefully designed training recipe, with large-scale high-quality CoT data, is effective to unlock strong reasoning capabilities even in resource-constrained small models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_21233 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math Xu, Haoran Peng, Baolin Awadalla, Hany Chen, Dongdong Chen, Yen-Chun Gao, Mei Kim, Young Jin Li, Yunsheng Ren, Liliang Shen, Yelong Wang, Shuohang Xu, Weijian Gao, Jianfeng Chen, Weizhu Computation and Language Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs) by training them to explicitly generate intermediate reasoning steps. While LLMs readily benefit from such techniques, improving reasoning in Small Language Models (SLMs) remains challenging due to their limited model capacity. Recent work by Deepseek-R1 demonstrates that distillation from LLM-generated synthetic data can substantially improve the reasoning ability of SLM. However, the detailed modeling recipe is not disclosed. In this work, we present a systematic training recipe for SLMs that consists of four steps: (1) large-scale mid-training on diverse distilled long-CoT data, (2) supervised fine-tuning on high-quality long-CoT data, (3) Rollout DPO leveraging a carefully curated preference dataset, and (4) Reinforcement Learning (RL) with Verifiable Reward. We apply our method on Phi-4-Mini, a compact 3.8B-parameter model. The resulting Phi-4-Mini-Reasoning model exceeds, on math reasoning tasks, much larger reasoning models, e.g., outperforming DeepSeek-R1-Distill-Qwen-7B by 3.2 points and DeepSeek-R1-Distill-Llama-8B by 7.7 points on Math-500. Our results validate that a carefully designed training recipe, with large-scale high-quality CoT data, is effective to unlock strong reasoning capabilities even in resource-constrained small models. |
| title | Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.21233 |