On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911658126868480 |
|---|---|
| author | Ye, Hao Dang, Jisheng Fang, Junfeng Wang, Bimei Zhang, Yizhou Lv, Ning Zhang, Wencan Peng, Hong Hu, Bin Chua, Tat-Seng |
| author_facet | Ye, Hao Dang, Jisheng Fang, Junfeng Wang, Bimei Zhang, Yizhou Lv, Ning Zhang, Wencan Peng, Hong Hu, Bin Chua, Tat-Seng |
| contents | Recent extensive research has demonstrated that the enhanced reasoning capabilities acquired by models through Reinforcement Learning with Verifiable Rewards (RLVR) are primarily concentrated within the rank-1 components. Predicated on this observation, we employed Periodic Rank-1 Substitution and identified a counterintuitive phenomenon: RLVR may exhibit implicit reward overfitting to the training dataset. Specifically, the model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process. Furthermore, we characterize three distinct properties of RL training: (1) The effective rank-1 component in RLVR don't maintain other model knowledge except mathematical reasoning capability. (2) RLVR fundamentally functions by optimizing a specific singular spectrum. The distribution of singular values of almost all linear layers in RLVR-trained model behaves like heavy-tailed distribution. (3) the left singular vectors associated with rank-1 components demonstrate a stronger alignment tendency during training, which echoes the discovery that RLVR is optimizing sampling efficiency in essence. Taken together, our findings and analysis further reveal how RLVR shapes model parameters and offer potential insights for improving existing RL paradigms or other training paradigms to implement continual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06523 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR Ye, Hao Dang, Jisheng Fang, Junfeng Wang, Bimei Zhang, Yizhou Lv, Ning Zhang, Wencan Peng, Hong Hu, Bin Chua, Tat-Seng Machine Learning Artificial Intelligence Recent extensive research has demonstrated that the enhanced reasoning capabilities acquired by models through Reinforcement Learning with Verifiable Rewards (RLVR) are primarily concentrated within the rank-1 components. Predicated on this observation, we employed Periodic Rank-1 Substitution and identified a counterintuitive phenomenon: RLVR may exhibit implicit reward overfitting to the training dataset. Specifically, the model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process. Furthermore, we characterize three distinct properties of RL training: (1) The effective rank-1 component in RLVR don't maintain other model knowledge except mathematical reasoning capability. (2) RLVR fundamentally functions by optimizing a specific singular spectrum. The distribution of singular values of almost all linear layers in RLVR-trained model behaves like heavy-tailed distribution. (3) the left singular vectors associated with rank-1 components demonstrate a stronger alignment tendency during training, which echoes the discovery that RLVR is optimizing sampling efficiency in essence. Taken together, our findings and analysis further reveal how RLVR shapes model parameters and offer potential insights for improving existing RL paradigms or other training paradigms to implement continual learning. |
| title | On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06523 |