Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.07534 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911106727936000 |
|---|---|
| author | Deng, Jia Chen, Jie Chen, Zhipeng Cheng, Daixuan Bai, Fei Zhang, Beichen Min, Yinqian Gao, Yanzipeng Zhao, Wayne Xin Wen, Ji-Rong |
| author_facet | Deng, Jia Chen, Jie Chen, Zhipeng Cheng, Daixuan Bai, Fei Zhang, Beichen Min, Yinqian Gao, Yanzipeng Zhao, Wayne Xin Wen, Ji-Rong |
| contents | Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). Unlike traditional RL approaches, RLVR leverages rule-based feedback to guide LLMs in generating and refining complex reasoning chains -- a process critically dependent on effective exploration strategies. While prior work has demonstrated RLVR's empirical success, the fundamental mechanisms governing LLMs' exploration behaviors remain underexplored. This technical report presents a systematic investigation of exploration capacities in RLVR, covering four main aspects: (1) exploration space shaping, where we develop quantitative metrics to characterize LLMs' capability boundaries; (2) entropy-performance exchange, analyzed across training stages, individual instances, and token-level patterns; and (3) RL performance optimization, examining methods to effectively translate exploration gains into measurable improvements. By unifying previously identified insights with new empirical evidence, this work aims to provide a foundational framework for advancing RLVR systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07534 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR Deng, Jia Chen, Jie Chen, Zhipeng Cheng, Daixuan Bai, Fei Zhang, Beichen Min, Yinqian Gao, Yanzipeng Zhao, Wayne Xin Wen, Ji-Rong Computation and Language Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). Unlike traditional RL approaches, RLVR leverages rule-based feedback to guide LLMs in generating and refining complex reasoning chains -- a process critically dependent on effective exploration strategies. While prior work has demonstrated RLVR's empirical success, the fundamental mechanisms governing LLMs' exploration behaviors remain underexplored. This technical report presents a systematic investigation of exploration capacities in RLVR, covering four main aspects: (1) exploration space shaping, where we develop quantitative metrics to characterize LLMs' capability boundaries; (2) entropy-performance exchange, analyzed across training stages, individual instances, and token-level patterns; and (3) RL performance optimization, examining methods to effectively translate exploration gains into measurable improvements. By unifying previously identified insights with new empirical evidence, this work aims to provide a foundational framework for advancing RLVR systems. |
| title | From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.07534 |