RL Fine-Tuning Heals OOD Forgetting in SFT
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910202252492800 |
|---|---|
| author | Jin, Hangzhan Luan, Sitao Ni, Tianwei Lyu, Sicheng Rabusseau, Guillaume Rabbany, Reihaneh Precup, Doina Hamdaqa, Mohammad |
| author_facet | Jin, Hangzhan Luan, Sitao Ni, Tianwei Lyu, Sicheng Rabusseau, Guillaume Rabbany, Reihaneh Precup, Doina Hamdaqa, Mohammad |
| contents | Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) is a standard post-training recipe for improving Large Language Models (LLM) reasoning, but why it works remains unclear. We revisit the common claim that ``SFT memorizes, RL generalizes'' through checkpoint-wise analyses of in-distribution (ID) and out-of-distribution (OOD) reasoning. We find that OOD performance often peaks early during SFT and then declines despite continued improvement in ID reasoning. RL typically does not surpass this early SFT peak; rather, it restores OOD capability lost during later SFT, and only from a bounded range of SFT checkpoints. Further spectral analysis shows that this forgetting-and-recovery pattern correlates with rotations of singular vectors, while singular values remain largely stable. These findings suggest a more precise view of post-training dynamics: SFT can forget, RL can recover, and controlling singular-vector rotation may improve OOD robustness. Code is available at \href{https://github.com/jinhangzhan/RL\_Heals\_SFT.git}{https://github.com/jinhangzhan/RL\_Heals\_SFT}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12235 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RL Fine-Tuning Heals OOD Forgetting in SFT Jin, Hangzhan Luan, Sitao Ni, Tianwei Lyu, Sicheng Rabusseau, Guillaume Rabbany, Reihaneh Precup, Doina Hamdaqa, Mohammad Machine Learning Artificial Intelligence Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) is a standard post-training recipe for improving Large Language Models (LLM) reasoning, but why it works remains unclear. We revisit the common claim that ``SFT memorizes, RL generalizes'' through checkpoint-wise analyses of in-distribution (ID) and out-of-distribution (OOD) reasoning. We find that OOD performance often peaks early during SFT and then declines despite continued improvement in ID reasoning. RL typically does not surpass this early SFT peak; rather, it restores OOD capability lost during later SFT, and only from a bounded range of SFT checkpoints. Further spectral analysis shows that this forgetting-and-recovery pattern correlates with rotations of singular vectors, while singular values remain largely stable. These findings suggest a more precise view of post-training dynamics: SFT can forget, RL can recover, and controlling singular-vector rotation may improve OOD robustness. Code is available at \href{https://github.com/jinhangzhan/RL\_Heals\_SFT.git}{https://github.com/jinhangzhan/RL\_Heals\_SFT}. |
| title | RL Fine-Tuning Heals OOD Forgetting in SFT |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12235 |