Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929363605258240 |
|---|---|
| author | Kim, Dohyeong Cho, Taehyun Han, Seungyub Chung, Hojun Lee, Kyungjae Oh, Songhwai |
| author_facet | Kim, Dohyeong Cho, Taehyun Han, Seungyub Chung, Hojun Lee, Kyungjae Oh, Songhwai |
| contents | The field of risk-constrained reinforcement learning (RCRL) has been developed to effectively reduce the likelihood of worst-case scenarios by explicitly handling risk-measure-based constraints. However, the nonlinearity of risk measures makes it challenging to achieve convergence and optimality. To overcome the difficulties posed by the nonlinearity, we propose a spectral risk measure-constrained RL algorithm, spectral-risk-constrained policy optimization (SRCPO), a bilevel optimization approach that utilizes the duality of spectral risk measures. In the bilevel optimization structure, the outer problem involves optimizing dual variables derived from the risk measures, while the inner problem involves finding an optimal policy given these dual variables. The proposed method, to the best of our knowledge, is the first to guarantee convergence to an optimum in the tabular setting. Furthermore, the proposed method has been evaluated on continuous control tasks and showed the best performance among other RCRL algorithms satisfying the constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18698 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees Kim, Dohyeong Cho, Taehyun Han, Seungyub Chung, Hojun Lee, Kyungjae Oh, Songhwai Machine Learning Artificial Intelligence The field of risk-constrained reinforcement learning (RCRL) has been developed to effectively reduce the likelihood of worst-case scenarios by explicitly handling risk-measure-based constraints. However, the nonlinearity of risk measures makes it challenging to achieve convergence and optimality. To overcome the difficulties posed by the nonlinearity, we propose a spectral risk measure-constrained RL algorithm, spectral-risk-constrained policy optimization (SRCPO), a bilevel optimization approach that utilizes the duality of spectral risk measures. In the bilevel optimization structure, the outer problem involves optimizing dual variables derived from the risk measures, while the inner problem involves finding an optimal policy given these dual variables. The proposed method, to the best of our knowledge, is the first to guarantee convergence to an optimum in the tabular setting. Furthermore, the proposed method has been evaluated on continuous control tasks and showed the best performance among other RCRL algorithms satisfying the constraints. |
| title | Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.18698 |