Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915447716184064 |
|---|---|
| author | Wang, Zizhao Wang, Caroline Xiao, Xuesu Zhu, Yuke Stone, Peter |
| author_facet | Wang, Zizhao Wang, Caroline Xiao, Xuesu Zhu, Yuke Stone, Peter |
| contents | Two desiderata of reinforcement learning (RL) algorithms are the ability to learn from relatively little experience and the ability to learn policies that generalize to a range of problem specifications. In factored state spaces, one approach towards achieving both goals is to learn state abstractions, which only keep the necessary variables for learning the tasks at hand. This paper introduces Causal Bisimulation Modeling (CBM), a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM leverages and improves implicit modeling to train a high-fidelity causal dynamics model that can be reused for all tasks in the same environment. Empirical validation on manipulation environments and Deepmind Control Suite reveals that CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones. Furthermore, the derived state abstractions allow a task learner to achieve near-oracle levels of sample efficiency and outperform baselines on all tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12497 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning Wang, Zizhao Wang, Caroline Xiao, Xuesu Zhu, Yuke Stone, Peter Artificial Intelligence Machine Learning Robotics I.2.9; I.2.8; I.2.6 Two desiderata of reinforcement learning (RL) algorithms are the ability to learn from relatively little experience and the ability to learn policies that generalize to a range of problem specifications. In factored state spaces, one approach towards achieving both goals is to learn state abstractions, which only keep the necessary variables for learning the tasks at hand. This paper introduces Causal Bisimulation Modeling (CBM), a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM leverages and improves implicit modeling to train a high-fidelity causal dynamics model that can be reused for all tasks in the same environment. Empirical validation on manipulation environments and Deepmind Control Suite reveals that CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones. Furthermore, the derived state abstractions allow a task learner to achieve near-oracle levels of sample efficiency and outperform baselines on all tasks. |
| title | Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning |
| topic | Artificial Intelligence Machine Learning Robotics I.2.9; I.2.8; I.2.6 |
| url | https://arxiv.org/abs/2401.12497 |