Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917532604039168 |
|---|---|
| author | Dou, Zhihao Zhao, Qinjian Wan, Zhongwei Zhang, Dinggen Wang, Weida Raiyan, Towsif Chen, Benteng Pan, Qingtao Ouyang, Yang Song, Chaoda Gao, Zhiqiang Zhang, Shufei Biswas, Sumon |
| author_facet | Dou, Zhihao Zhao, Qinjian Wan, Zhongwei Zhang, Dinggen Wang, Weida Raiyan, Towsif Chen, Benteng Pan, Qingtao Ouyang, Yang Song, Chaoda Gao, Zhiqiang Zhang, Shufei Biswas, Sumon |
| contents | Large language models (LLMs) demonstrate strong reasoning abilities via Chain-of-Thought (CoT), but their token-level generation encourages local decisions and lacks global planning, often leading to redundant or inaccurate reasoning. Existing methods, such as tree-based search and reinforcement learning (RL), attempt to address this issue but incur high computational costs and still struggle to produce reliable reasoning trajectories. To address these challenges, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly improve high-level planning and fine-grained CoT reasoning. Specifically, in the first stage, a given LLM is responsible for summarizing CoT reasoning into compact high-level guidance, which is then leveraged for supervised fine-tuning. Then, we introduce a guidance-aware reinforcement learning method that jointly optimizes the final output and the quality of guidance, enhancing reasoning effectiveness. We evaluate PTA-GRPO on ten reasoning benchmarks across mathematics and natural sciences, using five diverse base models spanning multiple data modalities. The results show that PTA-GRPO consistently delivers significant improvements across models and tasks, demonstrating strong effectiveness and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01833 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning Dou, Zhihao Zhao, Qinjian Wan, Zhongwei Zhang, Dinggen Wang, Weida Raiyan, Towsif Chen, Benteng Pan, Qingtao Ouyang, Yang Song, Chaoda Gao, Zhiqiang Zhang, Shufei Biswas, Sumon Artificial Intelligence Computation and Language Large language models (LLMs) demonstrate strong reasoning abilities via Chain-of-Thought (CoT), but their token-level generation encourages local decisions and lacks global planning, often leading to redundant or inaccurate reasoning. Existing methods, such as tree-based search and reinforcement learning (RL), attempt to address this issue but incur high computational costs and still struggle to produce reliable reasoning trajectories. To address these challenges, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly improve high-level planning and fine-grained CoT reasoning. Specifically, in the first stage, a given LLM is responsible for summarizing CoT reasoning into compact high-level guidance, which is then leveraged for supervised fine-tuning. Then, we introduce a guidance-aware reinforcement learning method that jointly optimizes the final output and the quality of guidance, enhancing reasoning effectiveness. We evaluate PTA-GRPO on ten reasoning benchmarks across mathematics and natural sciences, using five diverse base models spanning multiple data modalities. The results show that PTA-GRPO consistently delivers significant improvements across models and tasks, demonstrating strong effectiveness and generalization. |
| title | Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.01833 |