Mirror Descent for Stochastic Control Problems with Measure-valued Controls
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918128025337856 |
|---|---|
| author | Kerimkulov, Bekzhan Šiška, David Szpruch, Łukasz Zhang, Yufei |
| author_facet | Kerimkulov, Bekzhan Šiška, David Szpruch, Łukasz Zhang, Yufei |
| contents | This paper studies the convergence of the mirror descent algorithm for finite horizon stochastic control problems with measure-valued control processes. The control objective involves a convex regularisation function, denoted as $h$, with regularisation strength determined by the weight $τ\ge 0$. The setting covers regularised relaxed control problems. Under suitable conditions, we establish the relative smoothness and convexity of the control objective with respect to the Bregman divergence of $h$, and prove linear convergence of the algorithm for $τ=0$ and exponential convergence for $τ>0$. The results apply to common regularisers including relative entropy, $χ^2$-divergence, and entropic Wasserstein costs. This validates recent reinforcement learning heuristics that adding regularisation accelerates the convergence of gradient methods. The proof exploits careful regularity estimates of backward stochastic differential equations in the bounded mean oscillation norm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01198 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mirror Descent for Stochastic Control Problems with Measure-valued Controls Kerimkulov, Bekzhan Šiška, David Szpruch, Łukasz Zhang, Yufei Optimization and Control Numerical Analysis Probability 93E20, 49M05, 68Q25, 60H30 This paper studies the convergence of the mirror descent algorithm for finite horizon stochastic control problems with measure-valued control processes. The control objective involves a convex regularisation function, denoted as $h$, with regularisation strength determined by the weight $τ\ge 0$. The setting covers regularised relaxed control problems. Under suitable conditions, we establish the relative smoothness and convexity of the control objective with respect to the Bregman divergence of $h$, and prove linear convergence of the algorithm for $τ=0$ and exponential convergence for $τ>0$. The results apply to common regularisers including relative entropy, $χ^2$-divergence, and entropic Wasserstein costs. This validates recent reinforcement learning heuristics that adding regularisation accelerates the convergence of gradient methods. The proof exploits careful regularity estimates of backward stochastic differential equations in the bounded mean oscillation norm. |
| title | Mirror Descent for Stochastic Control Problems with Measure-valued Controls |
| topic | Optimization and Control Numerical Analysis Probability 93E20, 49M05, 68Q25, 60H30 |
| url | https://arxiv.org/abs/2401.01198 |