Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916990790139904 |
|---|---|
| author | Jiang, Zishang Han, Jinyi Li, Tingyun Wang, Xinyi Jiang, Sihang Liang, Jiaqing Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua |
| author_facet | Jiang, Zishang Han, Jinyi Li, Tingyun Wang, Xinyi Jiang, Sihang Liang, Jiaqing Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04140 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs Jiang, Zishang Han, Jinyi Li, Tingyun Wang, Xinyi Jiang, Sihang Liang, Jiaqing Dai, Zhaoqian Ma, Shuguang Yu, Fei Xiao, Yanghua Artificial Intelligence Computation and Language Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online. |
| title | Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.04140 |