Milestone-Guided Policy Learning for Long-Horizon Language Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911657101361152 |
|---|---|
| author | Wang, Zixuan Yan, Yuchen Li, Hongxing Pan, Teng Li, Dingming Zhang, Ruiqing Lu, Weiming Xiao, Jun Zhuang, Yueting Shen, Yongliang |
| author_facet | Wang, Zixuan Yan, Yuchen Li, Hongxing Pan, Teng Li, Dingming Zhang, Ruiqing Lu, Weiming Xiao, Jun Zhuang, Yueting Shen, Yongliang |
| contents | While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06078 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Milestone-Guided Policy Learning for Long-Horizon Language Agents Wang, Zixuan Yan, Yuchen Li, Hongxing Pan, Teng Li, Dingming Zhang, Ruiqing Lu, Weiming Xiao, Jun Zhuang, Yueting Shen, Yongliang Computation and Language Artificial Intelligence While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON. |
| title | Milestone-Guided Policy Learning for Long-Horizon Language Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06078 |