Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.12979 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912648037138432 |
|---|---|
| author | Fan, Wei Yao, Wenlin Li, Zheng Yao, Feng Liu, Xin Qiu, Liang Yin, Qingyu Song, Yangqiu Yin, Bing |
| author_facet | Fan, Wei Yao, Wenlin Li, Zheng Yao, Feng Liu, Xin Qiu, Liang Yin, Qingyu Song, Yangqiu Yin, Bing |
| contents | Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12979 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping Fan, Wei Yao, Wenlin Li, Zheng Yao, Feng Liu, Xin Qiu, Liang Yin, Qingyu Song, Yangqiu Yin, Bing Artificial Intelligence Computation and Language Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget. |
| title | DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.12979 |