GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911717773017088 |
|---|---|
| author | Hu, Yuelin Yu, Zhenbo Cheng, Zhengxue Liu, Wei Song, Li |
| author_facet | Hu, Yuelin Yu, Zhenbo Cheng, Zhengxue Liu, Wei Song, Li |
| contents | Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_26184 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training Hu, Yuelin Yu, Zhenbo Cheng, Zhengxue Liu, Wei Song, Li Machine Learning Artificial Intelligence Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead. |
| title | GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.26184 |