Feasible Policy Iteration for Safe Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915195557773312 |
|---|---|
| author | Yang, Yujie Zheng, Zhilong Li, Shengbo Eben Xu, Wei Liu, Jingjing Zhan, Xianyuan Zhang, Ya-Qin |
| author_facet | Yang, Yujie Zheng, Zhilong Li, Shengbo Eben Xu, Wei Liu, Jingjing Zhan, Xianyuan Zhang, Ya-Qin |
| contents | Safety is the priority concern when applying reinforcement learning (RL) algorithms to real-world control problems. While policy iteration provides a fundamental algorithm for standard RL, an analogous theoretical algorithm for safe RL remains absent. In this paper, we propose feasible policy iteration (FPI), the first foundational dynamic programming algorithm for safe RL. FPI alternates between policy evaluation, region identification and policy improvement. This follows actor-critic-scenery (ACS) framework where scenery refers to a feasibility function that represents a feasible region. A region-wise update rule is developed for the policy improvement step, which maximizes state-value function inside the feasible region and minimizes feasibility function outside it. With this update rule, FPI guarantees monotonic expansion of feasible region, monotonic improvement of state-value function, and geometric convergence to the optimal safe policy. Experimental results demonstrate that FPI achieves strictly zero constraint violation on low-dimensional tasks and outperforms existing methods in constraint adherence and reward performance on high-dimensional tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_08845 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Feasible Policy Iteration for Safe Reinforcement Learning Yang, Yujie Zheng, Zhilong Li, Shengbo Eben Xu, Wei Liu, Jingjing Zhan, Xianyuan Zhang, Ya-Qin Machine Learning Systems and Control Safety is the priority concern when applying reinforcement learning (RL) algorithms to real-world control problems. While policy iteration provides a fundamental algorithm for standard RL, an analogous theoretical algorithm for safe RL remains absent. In this paper, we propose feasible policy iteration (FPI), the first foundational dynamic programming algorithm for safe RL. FPI alternates between policy evaluation, region identification and policy improvement. This follows actor-critic-scenery (ACS) framework where scenery refers to a feasibility function that represents a feasible region. A region-wise update rule is developed for the policy improvement step, which maximizes state-value function inside the feasible region and minimizes feasibility function outside it. With this update rule, FPI guarantees monotonic expansion of feasible region, monotonic improvement of state-value function, and geometric convergence to the optimal safe policy. Experimental results demonstrate that FPI achieves strictly zero constraint violation on low-dimensional tasks and outperforms existing methods in constraint adherence and reward performance on high-dimensional tasks. |
| title | Feasible Policy Iteration for Safe Reinforcement Learning |
| topic | Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2304.08845 |