Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.02316 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909644772868096 |
|---|---|
| author | Celik, Onur Li, Zechu Blessing, Denis Li, Ge Palenicek, Daniel Peters, Jan Chalvatzaki, Georgia Neumann, Gerhard |
| author_facet | Celik, Onur Li, Zechu Blessing, Denis Li, Ge Palenicek, Daniel Peters, Jan Chalvatzaki, Georgia Neumann, Gerhard |
| contents | Maximum entropy reinforcement learning (MaxEnt-RL) has become the standard approach to RL due to its beneficial exploration properties. Traditionally, policies are parameterized using Gaussian distributions, which significantly limits their representational capacity. Diffusion-based policies offer a more expressive alternative, yet integrating them into MaxEnt-RL poses challenges-primarily due to the intractability of computing their marginal entropy. To overcome this, we propose Diffusion-Based Maximum Entropy RL (DIME). \emph{DIME} leverages recent advances in approximate inference with diffusion models to derive a lower bound on the maximum entropy objective. Additionally, we propose a policy iteration scheme that provably converges to the optimal diffusion policy. Our method enables the use of expressive diffusion-based policies while retaining the principled exploration benefits of MaxEnt-RL, significantly outperforming other diffusion-based methods on challenging high-dimensional control benchmarks. It is also competitive with state-of-the-art non-diffusion based RL methods while requiring fewer algorithmic design choices and smaller update-to-data ratios, reducing computational complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02316 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DIME:Diffusion-Based Maximum Entropy Reinforcement Learning Celik, Onur Li, Zechu Blessing, Denis Li, Ge Palenicek, Daniel Peters, Jan Chalvatzaki, Georgia Neumann, Gerhard Machine Learning Maximum entropy reinforcement learning (MaxEnt-RL) has become the standard approach to RL due to its beneficial exploration properties. Traditionally, policies are parameterized using Gaussian distributions, which significantly limits their representational capacity. Diffusion-based policies offer a more expressive alternative, yet integrating them into MaxEnt-RL poses challenges-primarily due to the intractability of computing their marginal entropy. To overcome this, we propose Diffusion-Based Maximum Entropy RL (DIME). \emph{DIME} leverages recent advances in approximate inference with diffusion models to derive a lower bound on the maximum entropy objective. Additionally, we propose a policy iteration scheme that provably converges to the optimal diffusion policy. Our method enables the use of expressive diffusion-based policies while retaining the principled exploration benefits of MaxEnt-RL, significantly outperforming other diffusion-based methods on challenging high-dimensional control benchmarks. It is also competitive with state-of-the-art non-diffusion based RL methods while requiring fewer algorithmic design choices and smaller update-to-data ratios, reducing computational complexity. |
| title | DIME:Diffusion-Based Maximum Entropy Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.02316 |