Reinforcement Learning in Non-Markovian Environments
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913233733943296 |
|---|---|
| author | Chandak, Siddharth Shah, Pratik Borkar, Vivek S Dodhia, Parth |
| author_facet | Chandak, Siddharth Shah, Pratik Borkar, Vivek S Dodhia, Parth |
| contents | Motivated by the novel paradigm developed by Van Roy and coauthors for reinforcement learning in arbitrary non-Markovian environments, we propose a related formulation and explicitly pin down the error caused by non-Markovianity of observations when the Q-learning algorithm is applied on this formulation. Based on this observation, we propose that the criterion for agent design should be to seek good approximations for certain conditional laws. Inspired by classical stochastic control, we show that our problem reduces to that of recursive computation of approximate sufficient statistics. This leads to an autoencoder-based scheme for agent design which is then numerically tested on partially observed reinforcement learning environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_01595 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning in Non-Markovian Environments Chandak, Siddharth Shah, Pratik Borkar, Vivek S Dodhia, Parth Systems and Control Machine Learning Motivated by the novel paradigm developed by Van Roy and coauthors for reinforcement learning in arbitrary non-Markovian environments, we propose a related formulation and explicitly pin down the error caused by non-Markovianity of observations when the Q-learning algorithm is applied on this formulation. Based on this observation, we propose that the criterion for agent design should be to seek good approximations for certain conditional laws. Inspired by classical stochastic control, we show that our problem reduces to that of recursive computation of approximate sufficient statistics. This leads to an autoencoder-based scheme for agent design which is then numerically tested on partially observed reinforcement learning environments. |
| title | Reinforcement Learning in Non-Markovian Environments |
| topic | Systems and Control Machine Learning |
| url | https://arxiv.org/abs/2211.01595 |