Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918449002840064 |
|---|---|
| author | Singh, Utsav Chakraborty, Souradip Suttle, Wesley A. Sadler, Brian M. Asher, Derrik E. Sahu, Anit Kumar Shah, Mubarak Namboodiri, Vinay P. Bedi, Amrit Singh |
| author_facet | Singh, Utsav Chakraborty, Souradip Suttle, Wesley A. Sadler, Brian M. Asher, Derrik E. Sahu, Anit Kumar Shah, Mubarak Namboodiri, Vinay P. Bedi, Amrit Singh |
| contents | Hierarchical reinforcement learning (HRL) enables agents to solve complex, long-horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundamental challenges: (i) non-stationarity caused by the evolving lower-level policy during training, which destabilizes higher-level learning, and (ii) the generation of infeasible subgoals that lower-level policies cannot achieve. To address these challenges, we introduce DIPPER, a novel HRL framework that formulates goal-conditioned HRL as a bi-level optimization problem and leverages direct preference optimization (DPO) to train the higher-level policy. By learning from stationary preference comparisons over subgoal sequences rather than rewards that depend on the evolving lower-level policy, DIPPER mitigates the impact of non-stationarity on hierarchical learning. To address infeasible subgoals, DIPPER incorporates lower-level value function regularization that encourages the higher-level policy to propose achievable subgoals. We also introduce two novel metrics to quantitatively verify that DIPPER mitigates non-stationarity and infeasible subgoal generation issues in HRL. We perform empirical evaluations on challenging robotic navigation and manipulation benchmarks and show that DIPPER achieves upto 40% improvements over state-of-the-art baselines, demonstrating that preference-based methods can effectively alleviate persistent challenges in hierarchical |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00361 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach Singh, Utsav Chakraborty, Souradip Suttle, Wesley A. Sadler, Brian M. Asher, Derrik E. Sahu, Anit Kumar Shah, Mubarak Namboodiri, Vinay P. Bedi, Amrit Singh Machine Learning Hierarchical reinforcement learning (HRL) enables agents to solve complex, long-horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundamental challenges: (i) non-stationarity caused by the evolving lower-level policy during training, which destabilizes higher-level learning, and (ii) the generation of infeasible subgoals that lower-level policies cannot achieve. To address these challenges, we introduce DIPPER, a novel HRL framework that formulates goal-conditioned HRL as a bi-level optimization problem and leverages direct preference optimization (DPO) to train the higher-level policy. By learning from stationary preference comparisons over subgoal sequences rather than rewards that depend on the evolving lower-level policy, DIPPER mitigates the impact of non-stationarity on hierarchical learning. To address infeasible subgoals, DIPPER incorporates lower-level value function regularization that encourages the higher-level policy to propose achievable subgoals. We also introduce two novel metrics to quantitatively verify that DIPPER mitigates non-stationarity and infeasible subgoal generation issues in HRL. We perform empirical evaluations on challenging robotic navigation and manipulation benchmarks and show that DIPPER achieves upto 40% improvements over state-of-the-art baselines, demonstrating that preference-based methods can effectively alleviate persistent challenges in hierarchical |
| title | Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.00361 |