On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910034468798464 |
|---|---|
| author | Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu |
| author_facet | Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu |
| contents | In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification Wu, Yongliang Zhou, Yizhou Ziheng, Zhou Peng, Yingzhe Ye, Xinyu Hu, Xinting Zhu, Wenbo Qi, Lu Yang, Ming-Hsuan Yang, Xu Machine Learning In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT. |
| title | On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.05629 |