PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909220980391936 |
|---|---|
| author | Chen, Chang Baek, Junyeob Deng, Fei Kawaguchi, Kenji Gulcehre, Caglar Ahn, Sungjin |
| author_facet | Chen, Chang Baek, Junyeob Deng, Fei Kawaguchi, Kenji Gulcehre, Caglar Ahn, Sungjin |
| contents | Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06793 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer Chen, Chang Baek, Junyeob Deng, Fei Kawaguchi, Kenji Gulcehre, Caglar Ahn, Sungjin Machine Learning Artificial Intelligence Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline \textit{value function learning}, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks. |
| title | PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.06793 |