ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915506648252416 |
|---|---|
| author | Zhang, Bonan Chen, Zhongqi Song, Bowen Li, Qinya Wu, Fan Chen, Guihai |
| author_facet | Zhang, Bonan Chen, Zhongqi Song, Bowen Li, Qinya Wu, Fan Chen, Guihai |
| contents | Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs) beyond pre-training and instruction tuning. A prominent line of work is RL with verifiable rewards (RLVR), which leverages automatically verifiable outcomes (e.g., correctness or executability) to generate reward signals. While efficient, this framework faces two key limitations: First, its binary feedback is too sparse to capture the quality of the reasoning process. Second, its coarse-grained rewards potentially lead to vanishing gradients. Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates. This joint design enriches the reward signal, providing finer-grained feedback and implicitly supervising the reasoning process. Experimental results demonstrate that our proposed method enhances RL performance across multiple datasets and reduces token consumption during inference, while incurring negligible additional training cost. Moreover, it can be used as a plug-in module to enhance other state-of-the-art RL methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17730 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs Zhang, Bonan Chen, Zhongqi Song, Bowen Li, Qinya Wu, Fan Chen, Guihai Machine Learning Computation and Language Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs) beyond pre-training and instruction tuning. A prominent line of work is RL with verifiable rewards (RLVR), which leverages automatically verifiable outcomes (e.g., correctness or executability) to generate reward signals. While efficient, this framework faces two key limitations: First, its binary feedback is too sparse to capture the quality of the reasoning process. Second, its coarse-grained rewards potentially lead to vanishing gradients. Inspired by observations from human learning, we introduce a RL technique that integrates verifiable outcomes with the model's own confidence estimates. This joint design enriches the reward signal, providing finer-grained feedback and implicitly supervising the reasoning process. Experimental results demonstrate that our proposed method enhances RL performance across multiple datasets and reduces token consumption during inference, while incurring negligible additional training cost. Moreover, it can be used as a plug-in module to enhance other state-of-the-art RL methods. |
| title | ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.17730 |