Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.11338 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914973190455296 |
|---|---|
| author | Park, Jaehyun Kim, Yunho Kim, Sejin Lee, Byung-Jun Kim, Sundong |
| author_facet | Park, Jaehyun Kim, Yunho Kim, Sejin Lee, Byung-Jun Kim, Sundong |
| contents | We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL: out-of-distribution samples and long-horizon problems. We leverage diffusion models to learn state-action sequence distributions and incorporate value functions for more balanced and adaptive decision-making. DIAR introduces an Adaptive Revaluation mechanism that dynamically adjusts decision lengths by comparing current and future state values, enabling flexible long-term decision-making. Furthermore, we address Q-value overestimation by combining Q-network learning with a value function guided by a diffusion model. The diffusion model generates diverse latent trajectories, enhancing policy robustness and generalization. As demonstrated in tasks like Maze2D, AntMaze, and Kitchen, DIAR consistently outperforms state-of-the-art algorithms in long-horizon, sparse-reward environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11338 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation Park, Jaehyun Kim, Yunho Kim, Sejin Lee, Byung-Jun Kim, Sundong Machine Learning Artificial Intelligence Robotics We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL: out-of-distribution samples and long-horizon problems. We leverage diffusion models to learn state-action sequence distributions and incorporate value functions for more balanced and adaptive decision-making. DIAR introduces an Adaptive Revaluation mechanism that dynamically adjusts decision lengths by comparing current and future state values, enabling flexible long-term decision-making. Furthermore, we address Q-value overestimation by combining Q-network learning with a value function guided by a diffusion model. The diffusion model generates diverse latent trajectories, enhancing policy robustness and generalization. As demonstrated in tasks like Maze2D, AntMaze, and Kitchen, DIAR consistently outperforms state-of-the-art algorithms in long-horizon, sparse-reward environments. |
| title | DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2410.11338 |