From Optimization to Control: Quasi Policy Iteration
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915705118523392 |
|---|---|
| author | Kolarijani, Mohammad Amin Sharifi Esfahani, Peyman Mohajerin |
| author_facet | Kolarijani, Mohammad Amin Sharifi Esfahani, Peyman Mohajerin |
| contents | Recent control algorithms for Markov decision processes (MDPs) have been designed using an implicit analogy with well-established optimization algorithms. In this paper, we adopt the quasi-Newton method (QNM) from convex optimization to introduce a novel control algorithm coined as quasi-policy iteration (QPI). In particular, QPI is based on a novel approximation of the ``Hessian'' matrix in the policy iteration algorithm, which exploits two linear structural constraints specific to MDPs and allows for the incorporation of prior information on the transition probability kernel. While the proposed algorithm has the same computational complexity as value iteration, it exhibits an empirical convergence behavior similar to that of QNM with a low sensitivity to the discount factor. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_11166 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | From Optimization to Control: Quasi Policy Iteration Kolarijani, Mohammad Amin Sharifi Esfahani, Peyman Mohajerin Optimization and Control Machine Learning Systems and Control Recent control algorithms for Markov decision processes (MDPs) have been designed using an implicit analogy with well-established optimization algorithms. In this paper, we adopt the quasi-Newton method (QNM) from convex optimization to introduce a novel control algorithm coined as quasi-policy iteration (QPI). In particular, QPI is based on a novel approximation of the ``Hessian'' matrix in the policy iteration algorithm, which exploits two linear structural constraints specific to MDPs and allows for the incorporation of prior information on the transition probability kernel. While the proposed algorithm has the same computational complexity as value iteration, it exhibits an empirical convergence behavior similar to that of QNM with a low sensitivity to the discount factor. |
| title | From Optimization to Control: Quasi Policy Iteration |
| topic | Optimization and Control Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2311.11166 |