A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909639726071808 |
|---|---|
| author | Kerimkulov, Bekzhan Leahy, James-Michael Siska, David Szpruch, Lukasz Zhang, Yufei |
| author_facet | Kerimkulov, Bekzhan Leahy, James-Michael Siska, David Szpruch, Lukasz Zhang, Yufei |
| contents | We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows. Our analysis provides a theoretical foundation for developing various discrete policy gradient algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02951 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces Kerimkulov, Bekzhan Leahy, James-Michael Siska, David Szpruch, Lukasz Zhang, Yufei Optimization and Control Machine Learning Probability 90C40, 93E20, 90C26, 60B05, 90C53 We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows. Our analysis provides a theoretical foundation for developing various discrete policy gradient algorithms. |
| title | A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces |
| topic | Optimization and Control Machine Learning Probability 90C40, 93E20, 90C26, 60B05, 90C53 |
| url | https://arxiv.org/abs/2310.02951 |