Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.21625 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912789193293824 |
|---|---|
| author | Tang, Xinyu Zhan, Yuliang Li, Zhixun Zhao, Wayne Xin Zhang, Zhenduo Wen, Zujie Zhang, Zhiqiang Zhou, Jun |
| author_facet | Tang, Xinyu Zhan, Yuliang Li, Zhixun Zhao, Wayne Xin Zhang, Zhenduo Wen, Zujie Zhang, Zhiqiang Zhou, Jun |
| contents | Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_21625 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards Tang, Xinyu Zhan, Yuliang Li, Zhixun Zhao, Wayne Xin Zhang, Zhenduo Wen, Zujie Zhang, Zhiqiang Zhou, Jun Computation and Language Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach. |
| title | Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.21625 |