Using Forwards-Backwards Models to Approximate MDP Homomorphisms
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916144005251072 |
|---|---|
| author | Mavor-Parker, Augustine N. Sargent, Matthew J. Pehle, Christian Banino, Andrea Griffin, Lewis D. Barry, Caswell |
| author_facet | Mavor-Parker, Augustine N. Sargent, Matthew J. Pehle, Christian Banino, Andrea Griffin, Lewis D. Barry, Caswell |
| contents | Reinforcement learning agents must painstakingly learn through trial and error what sets of state-action pairs are value equivalent -- requiring an often prohibitively large amount of environment experience. MDP homomorphisms have been proposed that reduce the MDP of an environment to an abstract MDP, enabling better sample efficiency. Consequently, impressive improvements have been achieved when a suitable homomorphism can be constructed a priori -- usually by exploiting a practitioner's knowledge of environment symmetries. We propose a novel approach to constructing homomorphisms in discrete action spaces, which uses a learnt model of environment dynamics to infer which state-action pairs lead to the same state -- which can reduce the size of the state-action space by a factor as large as the cardinality of the original action space. In MinAtar, we report an almost 4x improvement over a value-based off-policy baseline in the low sample limit, when averaging over all games and optimizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2209_06356 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Using Forwards-Backwards Models to Approximate MDP Homomorphisms Mavor-Parker, Augustine N. Sargent, Matthew J. Pehle, Christian Banino, Andrea Griffin, Lewis D. Barry, Caswell Machine Learning Artificial Intelligence Reinforcement learning agents must painstakingly learn through trial and error what sets of state-action pairs are value equivalent -- requiring an often prohibitively large amount of environment experience. MDP homomorphisms have been proposed that reduce the MDP of an environment to an abstract MDP, enabling better sample efficiency. Consequently, impressive improvements have been achieved when a suitable homomorphism can be constructed a priori -- usually by exploiting a practitioner's knowledge of environment symmetries. We propose a novel approach to constructing homomorphisms in discrete action spaces, which uses a learnt model of environment dynamics to infer which state-action pairs lead to the same state -- which can reduce the size of the state-action space by a factor as large as the cardinality of the original action space. In MinAtar, we report an almost 4x improvement over a value-based off-policy baseline in the low sample limit, when averaging over all games and optimizers. |
| title | Using Forwards-Backwards Models to Approximate MDP Homomorphisms |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2209.06356 |