Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913868453773312 |
|---|---|
| author | Yin, Huifeng Zhao, Yu Wu, Minghao Ni, Xuanfan Zeng, Bo Wang, Hao Shi, Tianqi Shao, Liangying Lyu, Chenyang Wang, Longyue Luo, Weihua Zhang, Kaifu |
| author_facet | Yin, Huifeng Zhao, Yu Wu, Minghao Ni, Xuanfan Zeng, Bo Wang, Hao Shi, Tianqi Shao, Liangying Lyu, Chenyang Wang, Longyue Luo, Weihua Zhang, Kaifu |
| contents | Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT). Distillation--post-training on LRMs-generated data--is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e. over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods. To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search(MCTS). We then exploit a set of CoT-aware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Post-training Objective, to enhance SFT and RL on the constructed data. We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME). instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking. The project homepage is https://github.com/AIDC-AI/Marco-o1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01461 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models Yin, Huifeng Zhao, Yu Wu, Minghao Ni, Xuanfan Zeng, Bo Wang, Hao Shi, Tianqi Shao, Liangying Lyu, Chenyang Wang, Longyue Luo, Weihua Zhang, Kaifu Machine Learning Artificial Intelligence Computation and Language Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT). Distillation--post-training on LRMs-generated data--is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e. over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods. To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search(MCTS). We then exploit a set of CoT-aware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Post-training Objective, to enhance SFT and RL on the constructed data. We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME). instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking. The project homepage is https://github.com/AIDC-AI/Marco-o1. |
| title | Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.01461 |