Sample-Efficient Constrained Reinforcement Learning with General Parameterization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913568234930176 |
|---|---|
| author | Mondal, Washim Uddin Aggarwal, Vaneet |
| author_facet | Mondal, Washim Uddin Aggarwal, Vaneet |
| contents | We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $ε$ global optimality gap and $ε$ constraint violation with $\tilde{\mathcal{O}}((1-γ)^{-7}ε^{-2})$ sample complexity for general parameterized policies where $γ$ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of $\mathcal{O}((1-γ)^{-1}ε^{-2})$ and achieves the theoretical lower bound in $ε^{-1}$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_10624 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sample-Efficient Constrained Reinforcement Learning with General Parameterization Mondal, Washim Uddin Aggarwal, Vaneet Machine Learning Artificial Intelligence We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $ε$ global optimality gap and $ε$ constraint violation with $\tilde{\mathcal{O}}((1-γ)^{-7}ε^{-2})$ sample complexity for general parameterized policies where $γ$ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of $\mathcal{O}((1-γ)^{-1}ε^{-2})$ and achieves the theoretical lower bound in $ε^{-1}$. |
| title | Sample-Efficient Constrained Reinforcement Learning with General Parameterization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.10624 |