Robust Offline Reinforcement learning with Heavy-Tailed Rewards
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914735162654720 |
|---|---|
| author | Zhu, Jin Wan, Runzhe Qi, Zhengling Luo, Shikai Shi, Chengchun |
| author_facet | Zhu, Jin Wan, Runzhe Qi, Zhengling Luo, Shikai Shi, Chengchun |
| contents | This paper endeavors to augment the robustness of offline reinforcement learning (RL) in scenarios laden with heavy-tailed rewards, a prevalent circumstance in real-world applications. We propose two algorithmic frameworks, ROAM and ROOM, for robust off-policy evaluation and offline policy optimization (OPO), respectively. Central to our frameworks is the strategic incorporation of the median-of-means method with offline RL, enabling straightforward uncertainty estimation for the value function estimator. This not only adheres to the principle of pessimism in OPO but also adeptly manages heavy-tailed rewards. Theoretical results and extensive experiments demonstrate that our two frameworks outperform existing methods on the logged dataset exhibits heavy-tailed reward distributions. The implementation of the proposal is available at https://github.com/Mamba413/ROOM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_18715 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Robust Offline Reinforcement learning with Heavy-Tailed Rewards Zhu, Jin Wan, Runzhe Qi, Zhengling Luo, Shikai Shi, Chengchun Machine Learning Artificial Intelligence This paper endeavors to augment the robustness of offline reinforcement learning (RL) in scenarios laden with heavy-tailed rewards, a prevalent circumstance in real-world applications. We propose two algorithmic frameworks, ROAM and ROOM, for robust off-policy evaluation and offline policy optimization (OPO), respectively. Central to our frameworks is the strategic incorporation of the median-of-means method with offline RL, enabling straightforward uncertainty estimation for the value function estimator. This not only adheres to the principle of pessimism in OPO but also adeptly manages heavy-tailed rewards. Theoretical results and extensive experiments demonstrate that our two frameworks outperform existing methods on the logged dataset exhibits heavy-tailed reward distributions. The implementation of the proposal is available at https://github.com/Mamba413/ROOM. |
| title | Robust Offline Reinforcement learning with Heavy-Tailed Rewards |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.18715 |