Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.15609 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911398821363712 |
|---|---|
| author | Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun |
| author_facet | Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) is a central paradigm for turning large language models (LLMs) into reliable problem solvers, especially in logic-heavy domains. Despite its empirical success, it remains unclear whether RLVR elicits novel capabilities or merely sharpens the distribution over existing knowledge. We study this by formalizing over-sharpening, a phenomenon where the policy collapses onto limited modes, suppressing valid alternatives. At a high level, we discover finite-batch updates intrinsically bias learning toward sampled modes, triggering a collapse that propagates globally via semantic coupling. To mitigate this, we propose inverse-success advantage calibration to prioritize difficult queries and distribution-level calibration to diversify sampling via a memory network. Empirical evaluations validate that our strategies can effectively improve generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15609 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun Machine Learning Computation and Language Reinforcement Learning with Verifiable Rewards (RLVR) is a central paradigm for turning large language models (LLMs) into reliable problem solvers, especially in logic-heavy domains. Despite its empirical success, it remains unclear whether RLVR elicits novel capabilities or merely sharpens the distribution over existing knowledge. We study this by formalizing over-sharpening, a phenomenon where the policy collapses onto limited modes, suppressing valid alternatives. At a high level, we discover finite-batch updates intrinsically bias learning toward sampled modes, triggering a collapse that propagates globally via semantic coupling. To mitigate this, we propose inverse-success advantage calibration to prioritize difficult queries and distribution-level calibration to diversify sampling via a memory network. Empirical evaluations validate that our strategies can effectively improve generalization. |
| title | When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.15609 |