Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912399216345088 |
|---|---|
| author | Wang, Vivienne Huiling Wang, Tinghuai Pajarinen, Joni |
| author_facet | Wang, Vivienne Huiling Wang, Tinghuai Pajarinen, Joni |
| contents | Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21750 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals Wang, Vivienne Huiling Wang, Tinghuai Pajarinen, Joni Machine Learning Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks. |
| title | Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.21750 |