Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.13316 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911846334726144 |
|---|---|
| author | Cho, Namkyeong Kim, Yeoneung |
| author_facet | Cho, Namkyeong Kim, Yeoneung |
| contents | We address the crucial yet underexplored stability properties of the Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning contexts, specifically for Lipschitz continuous optimal control problems. We bridge the gap between Lipschitz continuous optimal control problems and classical optimal control problems in the viscosity solutions framework, offering new insights into the stability of the value function of Lipschitz continuous optimal control problems. By introducing structural assumptions on the dynamics and reward functions, we further study the rate of convergence of value functions. Moreover, we introduce a generalized framework for Lipschitz continuous control problems that incorporates the original problem and leverage it to propose a new HJB-based reinforcement learning algorithm. The stability properties and performance of the proposed method are tested with well-known benchmark examples in comparison with existing approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13316 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the stability of Lipschitz continuous control problems and its application to reinforcement learning Cho, Namkyeong Kim, Yeoneung Optimization and Control Machine Learning Analysis of PDEs 49L25, 49L20 We address the crucial yet underexplored stability properties of the Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning contexts, specifically for Lipschitz continuous optimal control problems. We bridge the gap between Lipschitz continuous optimal control problems and classical optimal control problems in the viscosity solutions framework, offering new insights into the stability of the value function of Lipschitz continuous optimal control problems. By introducing structural assumptions on the dynamics and reward functions, we further study the rate of convergence of value functions. Moreover, we introduce a generalized framework for Lipschitz continuous control problems that incorporates the original problem and leverage it to propose a new HJB-based reinforcement learning algorithm. The stability properties and performance of the proposed method are tested with well-known benchmark examples in comparison with existing approaches. |
| title | On the stability of Lipschitz continuous control problems and its application to reinforcement learning |
| topic | Optimization and Control Machine Learning Analysis of PDEs 49L25, 49L20 |
| url | https://arxiv.org/abs/2404.13316 |