Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910045864722432 |
|---|---|
| author | Satheesh, Anirudh Barman, Pankaj Kumar Mondal, Washim Uddin Aggarwal, Vaneet |
| author_facet | Satheesh, Anirudh Barman, Pankaj Kumar Mondal, Washim Uddin Aggarwal, Vaneet |
| contents | We study infinite-horizon Constrained Markov Decision Processes (CMDPs) with general policy parameterizations and multi-layer neural network critics. Existing theoretical analyses for constrained reinforcement learning largely rely on tabular policies or linear critics, which limits their applicability to high-dimensional and continuous control problems. We propose a primal-dual natural actor-critic algorithm that integrates neural critic estimation with natural policy gradient updates and leverages Neural Tangent Kernel (NTK) theory to control function-approximation error under Markovian sampling, without requiring access to mixing-time oracles. We establish global convergence and cumulative constraint violation rates of $\tilde{\mathcal{O}}(T^-1/4)$ up to approximation errors induced by the policy and critic classes. Our results provide the first such guarantees for CMDPs with general policies and multi-layer neural critics, substantially extending the theoretical foundations of actor-critic methods beyond the linear-critic regime. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07698 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization Satheesh, Anirudh Barman, Pankaj Kumar Mondal, Washim Uddin Aggarwal, Vaneet Machine Learning We study infinite-horizon Constrained Markov Decision Processes (CMDPs) with general policy parameterizations and multi-layer neural network critics. Existing theoretical analyses for constrained reinforcement learning largely rely on tabular policies or linear critics, which limits their applicability to high-dimensional and continuous control problems. We propose a primal-dual natural actor-critic algorithm that integrates neural critic estimation with natural policy gradient updates and leverages Neural Tangent Kernel (NTK) theory to control function-approximation error under Markovian sampling, without requiring access to mixing-time oracles. We establish global convergence and cumulative constraint violation rates of $\tilde{\mathcal{O}}(T^-1/4)$ up to approximation errors induced by the policy and critic classes. Our results provide the first such guarantees for CMDPs with general policies and multi-layer neural critics, substantially extending the theoretical foundations of actor-critic methods beyond the linear-critic regime. |
| title | Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2603.07698 |