Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.13407 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915798091563008 |
|---|---|
| author | Zhao, Anhao Chen, Ziyang Tong, Junlong Fan, Yingqi Ye, Fanghua Li, Shuhao Ma, Yunpu Li, Wenjie Shen, Xiaoyu |
| author_facet | Zhao, Anhao Chen, Ziyang Tong, Junlong Fan, Yingqi Ye, Fanghua Li, Shuhao Ma, Yunpu Li, Wenjie Shen, Xiaoyu |
| contents | Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational cost. Recent methods add multi-reward objectives to jointly optimize correctness and brevity, but these complex extensions often destabilize training and yield suboptimal trade-offs. We revisit this objective and challenge the necessity of such complexity. Through principled analysis, we identify fundamental misalignments in this paradigm: KL regularization loses its intended role when correctness and length are directly verifiable, and group-wise normalization becomes ambiguous under multiple reward signals. By removing these two items and simplifying the reward to a truncation-based length penalty, we show that the optimization problem reduces to supervised fine-tuning on self-generated data filtered for both correctness and conciseness. We term this simplified training strategy on-policy SFT. Despite its simplicity, on-policy SFT consistently defines the accuracy-efficiency Pareto frontier. It reduces CoT length by up to 80 while maintaining original accuracy, surpassing more complex RL-based methods across five benchmarks. Furthermore, it significantly enhances training efficiency, reducing GPU memory usage by 50% and accelerating convergence by 70%. Our code is available at https://github.com/EIT-NLP/On-Policy-SFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13407 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On-Policy Supervised Fine-Tuning for Efficient Reasoning Zhao, Anhao Chen, Ziyang Tong, Junlong Fan, Yingqi Ye, Fanghua Li, Shuhao Ma, Yunpu Li, Wenjie Shen, Xiaoyu Artificial Intelligence Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational cost. Recent methods add multi-reward objectives to jointly optimize correctness and brevity, but these complex extensions often destabilize training and yield suboptimal trade-offs. We revisit this objective and challenge the necessity of such complexity. Through principled analysis, we identify fundamental misalignments in this paradigm: KL regularization loses its intended role when correctness and length are directly verifiable, and group-wise normalization becomes ambiguous under multiple reward signals. By removing these two items and simplifying the reward to a truncation-based length penalty, we show that the optimization problem reduces to supervised fine-tuning on self-generated data filtered for both correctness and conciseness. We term this simplified training strategy on-policy SFT. Despite its simplicity, on-policy SFT consistently defines the accuracy-efficiency Pareto frontier. It reduces CoT length by up to 80 while maintaining original accuracy, surpassing more complex RL-based methods across five benchmarks. Furthermore, it significantly enhances training efficiency, reducing GPU memory usage by 50% and accelerating convergence by 70%. Our code is available at https://github.com/EIT-NLP/On-Policy-SFT. |
| title | On-Policy Supervised Fine-Tuning for Efficient Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.13407 |