Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915389999415296 |
|---|---|
| author | Omura, Motoki Mukuta, Yusuke Ota, Kazuki Osa, Takayuki Harada, Tatsuya |
| author_facet | Omura, Motoki Mukuta, Yusuke Ota, Kazuki Osa, Takayuki Harada, Tatsuya |
| contents | Offline reinforcement learning (RL) aims to learn an optimal policy from a static dataset, making it particularly valuable in scenarios where data collection is costly, such as robotics. A major challenge in offline RL is distributional shift, where the learned policy deviates from the dataset distribution, potentially leading to unreliable out-of-distribution actions. To mitigate this issue, regularization techniques have been employed. While many existing methods utilize density ratio-based measures, such as the $f$-divergence, for regularization, we propose an approach that utilizes the Wasserstein distance, which is robust to out-of-distribution data and captures the similarity between actions. Our method employs input-convex neural networks (ICNNs) to model optimal transport maps, enabling the computation of the Wasserstein distance in a discriminator-free manner, thereby avoiding adversarial training and ensuring stable learning. Our approach demonstrates comparable or superior performance to widely used existing methods on the D4RL benchmark dataset. The code is available at https://github.com/motokiomura/Q-DOT . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_10843 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps Omura, Motoki Mukuta, Yusuke Ota, Kazuki Osa, Takayuki Harada, Tatsuya Machine Learning Artificial Intelligence Robotics Offline reinforcement learning (RL) aims to learn an optimal policy from a static dataset, making it particularly valuable in scenarios where data collection is costly, such as robotics. A major challenge in offline RL is distributional shift, where the learned policy deviates from the dataset distribution, potentially leading to unreliable out-of-distribution actions. To mitigate this issue, regularization techniques have been employed. While many existing methods utilize density ratio-based measures, such as the $f$-divergence, for regularization, we propose an approach that utilizes the Wasserstein distance, which is robust to out-of-distribution data and captures the similarity between actions. Our method employs input-convex neural networks (ICNNs) to model optimal transport maps, enabling the computation of the Wasserstein distance in a discriminator-free manner, thereby avoiding adversarial training and ensuring stable learning. Our approach demonstrates comparable or superior performance to widely used existing methods on the D4RL benchmark dataset. The code is available at https://github.com/motokiomura/Q-DOT . |
| title | Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2507.10843 |