Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.22648 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916048557572096 |
|---|---|
| author | Zeng, Xianzhou Huang, Jing Xie, Chunmei Nan, Gongrui Chen, Siye Lu, Mengyu Xiong, Weiqi Zhou, Qixuan Zhang, Junhao Zhu, Qiang Li, Yadong Xu, Xingzhong |
| author_facet | Zeng, Xianzhou Huang, Jing Xie, Chunmei Nan, Gongrui Chen, Siye Lu, Mengyu Xiong, Weiqi Zhou, Qixuan Zhang, Junhao Zhu, Qiang Li, Yadong Xu, Xingzhong |
| contents | The key to building trustworthy large language models (LLMs) lies in endowing them with inherent uncertainty expression capabilities, thereby mitigating overconfident errors in high-stakes applications. However, existing RL paradigms such as GRPO often suffer from Advantage Bias due to binary decision spaces and static uncertainty rewards, inducing either excessive conservatism or overconfidence. To tackle this challenge, this paper unveils the root causes of reward hacking and overconfidence in current RL paradigms incorporating uncertainty-based rewards, based on which we propose the UnCertainty-Aware Policy Optimization (UCPO) framework. UCPO employs Ternary Advantage Decoupling to separate and independently normalize deterministic and uncertain rollouts, thereby eliminating advantage bias. Furthermore, a Dynamic Uncertainty Reward Adjustment mechanism adapts uncertainty weights in real-time according to model evolution and instance difficulty. Experimental results in mathematical reasoning and general tasks demonstrate that UCPO effectively resolves the reward imbalance, significantly improving the reliability of the model beyond their knowledge boundaries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22648 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UCPO: Uncertainty-Aware Policy Optimization Zeng, Xianzhou Huang, Jing Xie, Chunmei Nan, Gongrui Chen, Siye Lu, Mengyu Xiong, Weiqi Zhou, Qixuan Zhang, Junhao Zhu, Qiang Li, Yadong Xu, Xingzhong Artificial Intelligence Machine Learning The key to building trustworthy large language models (LLMs) lies in endowing them with inherent uncertainty expression capabilities, thereby mitigating overconfident errors in high-stakes applications. However, existing RL paradigms such as GRPO often suffer from Advantage Bias due to binary decision spaces and static uncertainty rewards, inducing either excessive conservatism or overconfidence. To tackle this challenge, this paper unveils the root causes of reward hacking and overconfidence in current RL paradigms incorporating uncertainty-based rewards, based on which we propose the UnCertainty-Aware Policy Optimization (UCPO) framework. UCPO employs Ternary Advantage Decoupling to separate and independently normalize deterministic and uncertain rollouts, thereby eliminating advantage bias. Furthermore, a Dynamic Uncertainty Reward Adjustment mechanism adapts uncertainty weights in real-time according to model evolution and instance difficulty. Experimental results in mathematical reasoning and general tasks demonstrate that UCPO effectively resolves the reward imbalance, significantly improving the reliability of the model beyond their knowledge boundaries. |
| title | UCPO: Uncertainty-Aware Policy Optimization |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.22648 |