Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.05808 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915937612988416 |
|---|---|
| author | Zhen, Shuai Yu, Yanhua Guo, Ruopei Cheng, Nan Deng, Yang |
| author_facet | Zhen, Shuai Yu, Yanhua Guo, Ruopei Cheng, Nan Deng, Yang |
| contents | Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05808 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents Zhen, Shuai Yu, Yanhua Guo, Ruopei Cheng, Nan Deng, Yang Artificial Intelligence Machine Learning Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL. |
| title | Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.05808 |