Policy Gradient Algorithms in Average-Reward Multichain MDPs
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917283963600896 |
|---|---|
| author | Lee, Jongmin Ryu, Ernest K. |
| author_facet | Lee, Jongmin Ryu, Ernest K. |
| contents | While there is an extensive body of research analyzing policy gradient methods for discounted cumulative-reward MDPs, prior work on policy gradient methods for average-reward MDPs has been limited, with most existing results restricted to ergodic or unichain settings. In this work, we first establish a policy gradient theorem for average-reward multichain MDPs based on the invariance of the classification of recurrent and transient states. Building on this foundation, we develop refined analyses and obtain a collection of convergence and sample-complexity results that advance the understanding of this setting. In particular, we show that the proposed $α$-clipped policy mirror ascent algorithm attains an $ε$-optimal policy with respect to positive policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18003 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Policy Gradient Algorithms in Average-Reward Multichain MDPs Lee, Jongmin Ryu, Ernest K. Optimization and Control Computational Complexity While there is an extensive body of research analyzing policy gradient methods for discounted cumulative-reward MDPs, prior work on policy gradient methods for average-reward MDPs has been limited, with most existing results restricted to ergodic or unichain settings. In this work, we first establish a policy gradient theorem for average-reward multichain MDPs based on the invariance of the classification of recurrent and transient states. Building on this foundation, we develop refined analyses and obtain a collection of convergence and sample-complexity results that advance the understanding of this setting. In particular, we show that the proposed $α$-clipped policy mirror ascent algorithm attains an $ε$-optimal policy with respect to positive policies. |
| title | Policy Gradient Algorithms in Average-Reward Multichain MDPs |
| topic | Optimization and Control Computational Complexity |
| url | https://arxiv.org/abs/2602.18003 |