Convergence of actor-critic for entropy regularised MDPs in general action spaces
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915558675447808 |
|---|---|
| author | Zorba, Denis Šiška, David Szpruch, Lukasz |
| author_facet | Zorba, Denis Šiška, David Szpruch, Lukasz |
| contents | We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. We demonstrate stability and exponential convergence of the actor critic flow to the optimal policy. Finally, we address the interplay of the timescale separation and entropy regularisation and its effect on stability and convergence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14898 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Convergence of actor-critic for entropy regularised MDPs in general action spaces Zorba, Denis Šiška, David Szpruch, Lukasz Optimization and Control We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. We demonstrate stability and exponential convergence of the actor critic flow to the optimal policy. Finally, we address the interplay of the timescale separation and entropy regularisation and its effect on stability and convergence. |
| title | Convergence of actor-critic for entropy regularised MDPs in general action spaces |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2510.14898 |