Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.07017 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909681573691392 |
|---|---|
| author | Zheng, Tianyu Xing, Tianshun Gu, Qingshui Liang, Taoran Qu, Xingwei Zhou, Xin Li, Yizhi Wen, Zhoufutu Lin, Chenghua Huang, Wenhao Liu, Qian Zhang, Ge Ma, Zejun |
| author_facet | Zheng, Tianyu Xing, Tianshun Gu, Qingshui Liang, Taoran Qu, Xingwei Zhou, Xin Li, Yizhi Wen, Zhoufutu Lin, Chenghua Huang, Wenhao Liu, Qian Zhang, Ge Ma, Zejun |
| contents | Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07017 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | First Return, Entropy-Eliciting Explore Zheng, Tianyu Xing, Tianshun Gu, Qingshui Liang, Taoran Qu, Xingwei Zhou, Xin Li, Yizhi Wen, Zhoufutu Lin, Chenghua Huang, Wenhao Liu, Qian Zhang, Ge Ma, Zejun Artificial Intelligence Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration. |
| title | First Return, Entropy-Eliciting Explore |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.07017 |