Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914765590233088 |
|---|---|
| author | Wu, Lili Evans, Ben Islam, Riashat Seraj, Raihan Efroni, Yonathan Lamb, Alex |
| author_facet | Wu, Lili Evans, Ben Islam, Riashat Seraj, Raihan Efroni, Yonathan Lamb, Alex |
| contents | Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14552 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs Wu, Lili Evans, Ben Islam, Riashat Seraj, Raihan Efroni, Yonathan Lamb, Alex Machine Learning Artificial Intelligence Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly. |
| title | Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.14552 |