SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908419735158784 |
|---|---|
| author | Fu, Yuqian Chen, Tinghong Chai, Jiajun Wang, Xihuai Tu, Songjun Yin, Guojun Lin, Wei Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin |
| author_facet | Fu, Yuqian Chen, Tinghong Chai, Jiajun Wang, Xihuai Tu, Songjun Yin, Guojun Lin, Wei Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin |
| contents | Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_19767 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning Fu, Yuqian Chen, Tinghong Chai, Jiajun Wang, Xihuai Tu, Songjun Yin, Guojun Lin, Wei Zhang, Qichao Zhu, Yuanheng Zhao, Dongbin Computation and Language Artificial Intelligence Machine Learning Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks. |
| title | SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.19767 |