Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912217534824448 |
|---|---|
| author | Müller, Johannes Çaycı, Semih Montúfar, Guido |
| author_facet | Müller, Johannes Çaycı, Semih Montúfar, Guido |
| contents | Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19448 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients Müller, Johannes Çaycı, Semih Montúfar, Guido Optimization and Control Machine Learning Numerical Analysis Systems and Control 65K05, 90C05, 90C08, 90C40, 90C53 Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients. |
| title | Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients |
| topic | Optimization and Control Machine Learning Numerical Analysis Systems and Control 65K05, 90C05, 90C08, 90C40, 90C53 |
| url | https://arxiv.org/abs/2403.19448 |