Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.01103 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917482575429632 |
|---|---|
| author | Dong, Yiming Fu, Kun Li, Haoyu Zhu, Xinyuan Liu, Yurou Shao, Lijing Ye, Jieping Wang, Zheng |
| author_facet | Dong, Yiming Fu, Kun Li, Haoyu Zhu, Xinyuan Liu, Yurou Shao, Lijing Ye, Jieping Wang, Zheng |
| contents | Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01103 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Probing RLVR training instability through the lens of objective-level hacking Dong, Yiming Fu, Kun Li, Haoyu Zhu, Xinyuan Liu, Yurou Shao, Lijing Ye, Jieping Wang, Zheng Artificial Intelligence Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms. |
| title | Probing RLVR training instability through the lens of objective-level hacking |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.01103 |