Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910592763166720 |
|---|---|
| author | Huang, RenMing Liu, Shaochong Pei, Yunqiang Wang, Peng Wang, Guoqing Yang, Yang Shen, Hengtao |
| author_facet | Huang, RenMing Liu, Shaochong Pei, Yunqiang Wang, Peng Wang, Guoqing Yang, Yang Shen, Hengtao |
| contents | In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03996 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance Huang, RenMing Liu, Shaochong Pei, Yunqiang Wang, Peng Wang, Guoqing Yang, Yang Shen, Hengtao Machine Learning Robotics In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions. |
| title | Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance |
| topic | Machine Learning Robotics |
| url | https://arxiv.org/abs/2409.03996 |