On the Plasticity and Stability for Post-Training Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915779725754368 |
|---|---|
| author | Qiang, Wenwen Gu, Ziyin Zhou, Jiahuan Hu, Jie Wang, Jingyao Zheng, Changwen Xiong, Hui |
| author_facet | Qiang, Wenwen Gu, Ziyin Zhou, Jiahuan Hu, Jie Wang, Jingyao Zheng, Changwen Xiong, Hui |
| contents | Training stability remains a critical bottleneck for Group Relative Policy Optimization (GRPO), often manifesting as a trade-off between reasoning plasticity and general capability retention. We identify a root cause as the geometric conflict between plasticity and stability gradients, which leads to destructive interference. Crucially, we argue that deterministic projection methods are suboptimal for GRPO as they overlook the intrinsic stochasticity of group-based gradient estimates. To address this, we propose Probabilistic Conflict Resolution (PCR), a Bayesian framework that models gradients as random variables. PCR dynamically arbitrates conflicts via an uncertainty-aware ``soft projection'' mechanism, optimizing the signal-to-noise ratio. Extensive experiments demonstrate that PCR significantly smooths the training trajectory and achieves superior performance in various reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06453 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Plasticity and Stability for Post-Training Large Language Models Qiang, Wenwen Gu, Ziyin Zhou, Jiahuan Hu, Jie Wang, Jingyao Zheng, Changwen Xiong, Hui Machine Learning Training stability remains a critical bottleneck for Group Relative Policy Optimization (GRPO), often manifesting as a trade-off between reasoning plasticity and general capability retention. We identify a root cause as the geometric conflict between plasticity and stability gradients, which leads to destructive interference. Crucially, we argue that deterministic projection methods are suboptimal for GRPO as they overlook the intrinsic stochasticity of group-based gradient estimates. To address this, we propose Probabilistic Conflict Resolution (PCR), a Bayesian framework that models gradients as random variables. PCR dynamically arbitrates conflicts via an uncertainty-aware ``soft projection'' mechanism, optimizing the signal-to-noise ratio. Extensive experiments demonstrate that PCR significantly smooths the training trajectory and achieves superior performance in various reasoning tasks. |
| title | On the Plasticity and Stability for Post-Training Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.06453 |