Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.02896 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917115219410944 |
|---|---|
| author | Diaz, Gabriel Li, Lucky Zhang, Wenhao |
| author_facet | Diaz, Gabriel Li, Lucky Zhang, Wenhao |
| contents | Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entropy-regularized linear-quadratic (LQ) control problems with multiplicative noise over an infinite time horizon. First, we adapt the regularized policy gradient (RPG) algorithm to stochastic optimal control settings, proving that despite the non-convexity of the problem, RPG converges globally under conditions of gradient domination and almost-smoothness. Second, based on zero-order optimization approach, we introduce a novel model free RL algorithm: Sample-based regularized policy gradient (SB-RPG). SB-RPG operates without knowledge of system parameters yet still retains strong theoretical guarantees of global convergence. Our model leverages entropy regularization to address the exploration versus exploitation trade-off inherent in RL. Numerical simulations validate the theoretical results and demonstrate the efficiency of SB-RPG in unknown-parameters environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_02896 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise Diaz, Gabriel Li, Lucky Zhang, Wenhao Systems and Control Artificial Intelligence 37N35, 49N10 Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entropy-regularized linear-quadratic (LQ) control problems with multiplicative noise over an infinite time horizon. First, we adapt the regularized policy gradient (RPG) algorithm to stochastic optimal control settings, proving that despite the non-convexity of the problem, RPG converges globally under conditions of gradient domination and almost-smoothness. Second, based on zero-order optimization approach, we introduce a novel model free RL algorithm: Sample-based regularized policy gradient (SB-RPG). SB-RPG operates without knowledge of system parameters yet still retains strong theoretical guarantees of global convergence. Our model leverages entropy regularization to address the exploration versus exploitation trade-off inherent in RL. Numerical simulations validate the theoretical results and demonstrate the efficiency of SB-RPG in unknown-parameters environments. |
| title | Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise |
| topic | Systems and Control Artificial Intelligence 37N35, 49N10 |
| url | https://arxiv.org/abs/2510.02896 |