Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2509.26628 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914068460208128 |
|---|---|
| author | Liu, Runze Wang, Jiakang Shi, Yuling Xie, Zhihui An, Chenxin Zhang, Kaiyan Zhao, Jian Gu, Xiaodong Lin, Lei Hu, Wenping Li, Xiu Zhang, Fuzheng Zhou, Guorui Gai, Kun |
| author_facet | Liu, Runze Wang, Jiakang Shi, Yuling Xie, Zhihui An, Chenxin Zhang, Kaiyan Zhao, Jian Gu, Xiaodong Lin, Lei Hu, Wenping Li, Xiu Zhang, Fuzheng Zhou, Guorui Gai, Kun |
| contents | Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26628 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models Liu, Runze Wang, Jiakang Shi, Yuling Xie, Zhihui An, Chenxin Zhang, Kaiyan Zhao, Jian Gu, Xiaodong Lin, Lei Hu, Wenping Li, Xiu Zhang, Fuzheng Zhou, Guorui Gai, Kun Machine Learning Computation and Language Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency. |
| title | Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.26628 |