Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918036430127104 |
|---|---|
| author | Hu, Xiao Lu, Xingyu Mao, Liyuan Zhang, YiFan Zhang, Tianke Wen, Bin Yang, Fan Gao, Tingting Zhou, Guorui |
| author_facet | Hu, Xiao Lu, Xingyu Mao, Liyuan Zhang, YiFan Zhang, Tianke Wen, Bin Yang, Fan Gao, Tingting Zhou, Guorui |
| contents | Reinforcement learning (RL) has played an important role in improving the reasoning ability of large language models (LLMs). Some studies apply RL directly to \textit{smaller} base models (known as zero-RL) and also achieve notable progress. However, in this paper, we show that using only 920 examples, a simple distillation method based on the base model can clearly outperform zero-RL, which typically requires much more data and computational cost. By analyzing the token frequency in model outputs, we find that the distilled model shows more flexible reasoning. It uses anthropomorphic tokens and logical connectors much more often than the zero-RL model. Further analysis reveals that distillation enhances the presence of two advanced cognitive behaviors: Multi-Perspective Thinking or Attempting and Metacognitive Awareness. Frequent occurrences of these two advanced cognitive behaviors give rise to flexible reasoning, which is essential for solving complex reasoning problems, while zero-RL fails to significantly boost the frequency of these behaviors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21067 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning Hu, Xiao Lu, Xingyu Mao, Liyuan Zhang, YiFan Zhang, Tianke Wen, Bin Yang, Fan Gao, Tingting Zhou, Guorui Artificial Intelligence Reinforcement learning (RL) has played an important role in improving the reasoning ability of large language models (LLMs). Some studies apply RL directly to \textit{smaller} base models (known as zero-RL) and also achieve notable progress. However, in this paper, we show that using only 920 examples, a simple distillation method based on the base model can clearly outperform zero-RL, which typically requires much more data and computational cost. By analyzing the token frequency in model outputs, we find that the distilled model shows more flexible reasoning. It uses anthropomorphic tokens and logical connectors much more often than the zero-RL model. Further analysis reveals that distillation enhances the presence of two advanced cognitive behaviors: Multi-Perspective Thinking or Attempting and Metacognitive Awareness. Frequent occurrences of these two advanced cognitive behaviors give rise to flexible reasoning, which is essential for solving complex reasoning problems, while zero-RL fails to significantly boost the frequency of these behaviors. |
| title | Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.21067 |