Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.07085 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914174255235072 |
|---|---|
| author | Ashlag, Yonatan Koren, Uri Mutti, Mirco Derman, Esther Bacon, Pierre-Luc Mannor, Shie |
| author_facet | Ashlag, Yonatan Koren, Uri Mutti, Mirco Derman, Esther Bacon, Pierre-Luc Mannor, Shie |
| contents | State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_07085 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | State Entropy Regularization for Robust Reinforcement Learning Ashlag, Yonatan Koren, Uri Mutti, Mirco Derman, Esther Bacon, Pierre-Luc Mannor, Shie Machine Learning State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation. |
| title | State Entropy Regularization for Robust Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.07085 |