Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910998857777152 |
|---|---|
| author | Zhang, Kongcheng Yao, Qi Liu, Shunyu Wang, Yingjie Lai, Baisheng Ye, Jieping Song, Mingli Tao, Dacheng |
| author_facet | Zhang, Kongcheng Yao, Qi Liu, Shunyu Wang, Yingjie Lai, Baisheng Ye, Jieping Song, Mingli Tao, Dacheng |
| contents | Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08745 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning Zhang, Kongcheng Yao, Qi Liu, Shunyu Wang, Yingjie Lai, Baisheng Ye, Jieping Song, Mingli Tao, Dacheng Artificial Intelligence Computation and Language Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo. |
| title | Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.08745 |