Operator World Models for Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917823677202432 |
|---|---|
| author | Novelli, Pietro Pratticò, Marco Pontil, Massimiliano Ciliberto, Carlo |
| author_facet | Novelli, Pietro Pratticò, Marco Pontil, Massimiliano Ciliberto, Carlo |
| contents | Policy Mirror Descent (PMD) is a powerful and theoretically sound methodology for sequential decision-making. However, it is not directly applicable to Reinforcement Learning (RL) due to the inaccessibility of explicit action-value functions. We address this challenge by introducing a novel approach based on learning a world model of the environment using conditional mean embeddings. Leveraging tools from operator theory we derive a closed-form expression of the action-value function in terms of the world model via simple matrix operations. Combining these estimators with PMD leads to POWR, a new RL algorithm for which we prove convergence rates to the global optimum. Preliminary experiments in finite and infinite state settings support the effectiveness of our method |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19861 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Operator World Models for Reinforcement Learning Novelli, Pietro Pratticò, Marco Pontil, Massimiliano Ciliberto, Carlo Machine Learning Optimization and Control Policy Mirror Descent (PMD) is a powerful and theoretically sound methodology for sequential decision-making. However, it is not directly applicable to Reinforcement Learning (RL) due to the inaccessibility of explicit action-value functions. We address this challenge by introducing a novel approach based on learning a world model of the environment using conditional mean embeddings. Leveraging tools from operator theory we derive a closed-form expression of the action-value function in terms of the world model via simple matrix operations. Combining these estimators with PMD leads to POWR, a new RL algorithm for which we prove convergence rates to the global optimum. Preliminary experiments in finite and infinite state settings support the effectiveness of our method |
| title | Operator World Models for Reinforcement Learning |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2406.19861 |