Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916756374683648 |
|---|---|
| author | Davey, Ashley Zheng, Harry |
| author_facet | Davey, Ashley Zheng, Harry |
| contents | We prove convergence of the proximal policy gradient method for a class of constrained stochastic control problems with control in both the drift and diffusion of the state process. The problem requires either the running or terminal cost to be strongly convex, but other terms may be non-convex. The inclusion of control-dependent diffusion introduces additional complexity in regularity analysis of the associated backward stochastic differential equation. We provide sufficient conditions under which the control iterates converge linearly to the optimal control, by deriving representations and estimates of solutions to the adjoint backward stochastic differential equations. We introduce numerical algorithms that implement this method using deep learning and ordinary differential equation based techniques. These approaches enable high accuracy and scalability for stochastic control problems in higher dimensions. We provide numerical examples to demonstrate the accuracy and validate the theoretical convergence guarantees of the algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18379 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients Davey, Ashley Zheng, Harry Optimization and Control 68Q25, 93E20, 49M05, 35C05, 65C30 We prove convergence of the proximal policy gradient method for a class of constrained stochastic control problems with control in both the drift and diffusion of the state process. The problem requires either the running or terminal cost to be strongly convex, but other terms may be non-convex. The inclusion of control-dependent diffusion introduces additional complexity in regularity analysis of the associated backward stochastic differential equation. We provide sufficient conditions under which the control iterates converge linearly to the optimal control, by deriving representations and estimates of solutions to the adjoint backward stochastic differential equations. We introduce numerical algorithms that implement this method using deep learning and ordinary differential equation based techniques. These approaches enable high accuracy and scalability for stochastic control problems in higher dimensions. We provide numerical examples to demonstrate the accuracy and validate the theoretical convergence guarantees of the algorithms. |
| title | Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients |
| topic | Optimization and Control 68Q25, 93E20, 49M05, 35C05, 65C30 |
| url | https://arxiv.org/abs/2505.18379 |