Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912382504140800 |
|---|---|
| author | Liu, Xiaoyuan Liang, Tian He, Zhiwei Xu, Jiahao Wang, Wenxuan He, Pinjia Tu, Zhaopeng Mi, Haitao Yu, Dong |
| author_facet | Liu, Xiaoyuan Liang, Tian He, Zhiwei Xu, Jiahao Wang, Wenxuan He, Pinjia Tu, Zhaopeng Mi, Haitao Yu, Dong |
| contents | Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13445 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards Liu, Xiaoyuan Liang, Tian He, Zhiwei Xu, Jiahao Wang, Wenxuan He, Pinjia Tu, Zhaopeng Mi, Haitao Yu, Dong Artificial Intelligence Computation and Language Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners. |
| title | Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2505.13445 |