DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2020
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909476433428480 |
|---|---|
| author | Shrestha, Aayam Lee, Stefan Tadepalli, Prasad Fern, Alan |
| author_facet | Shrestha, Aayam Lee, Stefan Tadepalli, Prasad Fern, Alan |
| contents | We study an approach to offline reinforcement learning (RL) based on optimally solving finitely-represented MDPs derived from a static dataset of experience. This approach can be applied on top of any learned representation and has the potential to easily support multiple solution objectives as well as zero-shot adjustment to changing environments and goals. Our main contribution is to introduce the Deep Averagers with Costs MDP (DAC-MDP) and to investigate its solutions for offline RL. DAC-MDPs are a non-parametric model that can leverage deep representations and account for limited data by introducing costs for exploiting under-represented parts of the model. In theory, we show conditions that allow for lower-bounding the performance of DAC-MDP solutions. We also investigate the empirical behavior in a number of environments, including those with image-based observations. Overall, the experiments demonstrate that the framework can work in practice and scale to large complex offline RL problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2010_08891 |
| institution | arXiv |
| publishDate | 2020 |
| record_format | arxiv |
| spellingShingle | DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs Shrestha, Aayam Lee, Stefan Tadepalli, Prasad Fern, Alan Machine Learning Artificial Intelligence We study an approach to offline reinforcement learning (RL) based on optimally solving finitely-represented MDPs derived from a static dataset of experience. This approach can be applied on top of any learned representation and has the potential to easily support multiple solution objectives as well as zero-shot adjustment to changing environments and goals. Our main contribution is to introduce the Deep Averagers with Costs MDP (DAC-MDP) and to investigate its solutions for offline RL. DAC-MDPs are a non-parametric model that can leverage deep representations and account for limited data by introducing costs for exploiting under-represented parts of the model. In theory, we show conditions that allow for lower-bounding the performance of DAC-MDP solutions. We also investigate the empirical behavior in a number of environments, including those with image-based observations. Overall, the experiments demonstrate that the framework can work in practice and scale to large complex offline RL problems. |
| title | DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2010.08891 |