Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.05629 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910034468798464 |
|---|---|
| author | Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu |
| author_facet | Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu |
| contents | In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu Machine Learning In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT. |
| title | On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.05629 |