Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.05053 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917421856587776 |
|---|---|
| author | Zhao, Ziqi Ren, Zhaochun Zou, Jiahong Yang, Liu Xu, Zhiwei Ge, Xuri Chen, Zhumin Ma, Xinyu Shi, Daiting Wang, Shuaiqiang Yin, Dawei Xin, Xin |
| author_facet | Zhao, Ziqi Ren, Zhaochun Zou, Jiahong Yang, Liu Xu, Zhiwei Ge, Xuri Chen, Zhumin Ma, Xinyu Shi, Daiting Wang, Shuaiqiang Yin, Dawei Xin, Xin |
| contents | Reinforcement learning with verifiable rewards (RLVR) has proven effective in enhancing the reasoning of large language models (LLMs). Monte Carlo Tree Search (MCTS)-based extensions improve upon vanilla RLVR (e.g., GRPO) by providing tree-based reasoning rollouts that enable fine-grained and segment-level credit assignment. However, existing methods still suffer from limited exploration diversity and inefficient reasoning. To address the above challenges, we propose reinforced efficient reasoning via semantically diverse explorations, i.e., ROSE, for LLMs. To encourage more diverse reasoning exploration, our method incorporates a semantic-entropy-based branching strategy and an $\varepsilon$-exploration mechanism. The former operates on already sampled reasoning rollouts to capture semantic uncertainty and select branching points with high semantic divergence to generate new successive reasoning paths, whereas the latter stochastically initiates reasoning rollouts from the root, preventing the search process from becoming overly local. To improve efficiency, we design a length-aware segment-level advantage estimator that rewards concise and correct reasoning while penalizing unnecessarily long reasoning chains. Extensive experiments on various mathematical reasoning benchmarks with Qwen and Llama models validate the effectiveness and efficiency of ROSE. Codes are available at https://github.com/ZiqiZhao1/ROSE-rl. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05053 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reinforced Efficient Reasoning via Semantically Diverse Exploration Zhao, Ziqi Ren, Zhaochun Zou, Jiahong Yang, Liu Xu, Zhiwei Ge, Xuri Chen, Zhumin Ma, Xinyu Shi, Daiting Wang, Shuaiqiang Yin, Dawei Xin, Xin Artificial Intelligence Computation and Language Reinforcement learning with verifiable rewards (RLVR) has proven effective in enhancing the reasoning of large language models (LLMs). Monte Carlo Tree Search (MCTS)-based extensions improve upon vanilla RLVR (e.g., GRPO) by providing tree-based reasoning rollouts that enable fine-grained and segment-level credit assignment. However, existing methods still suffer from limited exploration diversity and inefficient reasoning. To address the above challenges, we propose reinforced efficient reasoning via semantically diverse explorations, i.e., ROSE, for LLMs. To encourage more diverse reasoning exploration, our method incorporates a semantic-entropy-based branching strategy and an $\varepsilon$-exploration mechanism. The former operates on already sampled reasoning rollouts to capture semantic uncertainty and select branching points with high semantic divergence to generate new successive reasoning paths, whereas the latter stochastically initiates reasoning rollouts from the root, preventing the search process from becoming overly local. To improve efficiency, we design a length-aware segment-level advantage estimator that rewards concise and correct reasoning while penalizing unnecessarily long reasoning chains. Extensive experiments on various mathematical reasoning benchmarks with Qwen and Llama models validate the effectiveness and efficiency of ROSE. Codes are available at https://github.com/ZiqiZhao1/ROSE-rl. |
| title | Reinforced Efficient Reasoning via Semantically Diverse Exploration |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2601.05053 |