Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.03018 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916016879042560 |
|---|---|
| author | Wang, Yutong Ji, Pengliang Li, Kaixin Bi, Baolong Feng, Tao Sartoretti, Guillaume |
| author_facet | Wang, Yutong Ji, Pengliang Li, Kaixin Bi, Baolong Feng, Tao Sartoretti, Guillaume |
| contents | Large Language Reasoning Models have demonstrated remarkable success on static tasks, yet their application to multi-round agentic planning in interactive environments faces two fundamental challenges. First, the intractable credit assignment problem renders conventional reinforcement learning ineffective in sparse-reward settings. Second, the computational overhead of verbose, step-by-step reasoning histories is prohibitive. To address these challenges, we propose BPO, a three-stage framework (bootstrapping, extrapolation, and refinement) that establishes a self-improving data flywheel to develop robust reasoning models for long-horizon, sparse-reward environments. Our framework first bootstraps efficient reasoning using the proposed planning quaternions with long-short chain-of-thought fusion. It then extrapolates to out-of-distribution tasks through complexity-stratified curriculum learning. Finally, the model iteratively refines itself by learning exclusively on experiences selected via reward-gated rejection sampling. Experiments on ALFWorld, ScienceWorld, and WebShop demonstrate that our approach achieves state-of-the-art with significant token efficiency, providing a new recipe for reasoning models in agentic planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03018 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning Wang, Yutong Ji, Pengliang Li, Kaixin Bi, Baolong Feng, Tao Sartoretti, Guillaume Artificial Intelligence Robotics Large Language Reasoning Models have demonstrated remarkable success on static tasks, yet their application to multi-round agentic planning in interactive environments faces two fundamental challenges. First, the intractable credit assignment problem renders conventional reinforcement learning ineffective in sparse-reward settings. Second, the computational overhead of verbose, step-by-step reasoning histories is prohibitive. To address these challenges, we propose BPO, a three-stage framework (bootstrapping, extrapolation, and refinement) that establishes a self-improving data flywheel to develop robust reasoning models for long-horizon, sparse-reward environments. Our framework first bootstraps efficient reasoning using the proposed planning quaternions with long-short chain-of-thought fusion. It then extrapolates to out-of-distribution tasks through complexity-stratified curriculum learning. Finally, the model iteratively refines itself by learning exclusively on experiences selected via reward-gated rejection sampling. Experiments on ALFWorld, ScienceWorld, and WebShop demonstrate that our approach achieves state-of-the-art with significant token efficiency, providing a new recipe for reasoning models in agentic planning. |
| title | Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning |
| topic | Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2508.03018 |