Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915253581774848 |
|---|---|
| author | Ma, Jin Wang, Gaozhan Zhang, Jianfeng |
| author_facet | Ma, Jin Wang, Gaozhan Zhang, Jianfeng |
| contents | In this paper we investigate the convergence of the Policy Iteration Algorithm (PIA) for a class of general continuous-time entropy-regularized stochastic control problems. In particular, instead of employing sophisticated PDE estimates for the iterative PDEs involved in the algorithm (see, e.g., Huang-Wang-Zhou(2025)), we shall provide a simple proof from scratch for the convergence of the PIA. Our approach builds on probabilistic representation formulae for solutions of PDEs and their derivatives. Moreover, in the finite horizon model and in the infinite horizon model with large discount factor, the similar arguments lead to a super-exponential rate of convergence without tear. Finally, with some extra efforts we show that our approach can be extended to the diffusion control case in the one dimensional setting, also with a super-exponential rate of convergence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10959 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach Ma, Jin Wang, Gaozhan Zhang, Jianfeng Optimization and Control Machine Learning 93E35, 60H30, 35Q93 In this paper we investigate the convergence of the Policy Iteration Algorithm (PIA) for a class of general continuous-time entropy-regularized stochastic control problems. In particular, instead of employing sophisticated PDE estimates for the iterative PDEs involved in the algorithm (see, e.g., Huang-Wang-Zhou(2025)), we shall provide a simple proof from scratch for the convergence of the PIA. Our approach builds on probabilistic representation formulae for solutions of PDEs and their derivatives. Moreover, in the finite horizon model and in the infinite horizon model with large discount factor, the similar arguments lead to a super-exponential rate of convergence without tear. Finally, with some extra efforts we show that our approach can be extended to the diffusion control case in the one dimensional setting, also with a super-exponential rate of convergence. |
| title | Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach |
| topic | Optimization and Control Machine Learning 93E35, 60H30, 35Q93 |
| url | https://arxiv.org/abs/2406.10959 |