Rethinking State Disentanglement in Causal Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912000881197056 |
|---|---|
| author | Cao, Haiyao Zhang, Zhen Cai, Panpan Liu, Yuhang Zou, Jinan Abbasnejad, Ehsan Huang, Biwei Gong, Mingming Hengel, Anton van den Shi, Javen Qinfeng |
| author_facet | Cao, Haiyao Zhang, Zhen Cai, Panpan Liu, Yuhang Zou, Jinan Abbasnejad, Ehsan Huang, Biwei Gong, Mingming Hengel, Anton van den Shi, Javen Qinfeng |
| contents | One of the significant challenges in reinforcement learning (RL) when dealing with noise is estimating latent states from observations. Causality provides rigorous theoretical support for ensuring that the underlying states can be uniquely recovered through identifiability. Consequently, some existing work focuses on establishing identifiability from a causal perspective to aid in the design of algorithms. However, these results are often derived from a purely causal viewpoint, which may overlook the specific RL context. We revisit this research line and find that incorporating RL-specific context can reduce unnecessary assumptions in previous identifiability analyses for latent states. More importantly, removing these assumptions allows algorithm design to go beyond the earlier boundaries constrained by them. Leveraging these insights, we propose a novel approach for general partially observable Markov Decision Processes (POMDPs) by replacing the complicated structural constraints in previous methods with two simple constraints for transition and reward preservation. With the two constraints, the proposed algorithm is guaranteed to disentangle state and noise that is faithful to the underlying dynamics. Empirical evidence from extensive benchmark control tasks demonstrates the superiority of our approach over existing counterparts in effectively disentangling state belief from noise. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13498 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rethinking State Disentanglement in Causal Reinforcement Learning Cao, Haiyao Zhang, Zhen Cai, Panpan Liu, Yuhang Zou, Jinan Abbasnejad, Ehsan Huang, Biwei Gong, Mingming Hengel, Anton van den Shi, Javen Qinfeng Machine Learning One of the significant challenges in reinforcement learning (RL) when dealing with noise is estimating latent states from observations. Causality provides rigorous theoretical support for ensuring that the underlying states can be uniquely recovered through identifiability. Consequently, some existing work focuses on establishing identifiability from a causal perspective to aid in the design of algorithms. However, these results are often derived from a purely causal viewpoint, which may overlook the specific RL context. We revisit this research line and find that incorporating RL-specific context can reduce unnecessary assumptions in previous identifiability analyses for latent states. More importantly, removing these assumptions allows algorithm design to go beyond the earlier boundaries constrained by them. Leveraging these insights, we propose a novel approach for general partially observable Markov Decision Processes (POMDPs) by replacing the complicated structural constraints in previous methods with two simple constraints for transition and reward preservation. With the two constraints, the proposed algorithm is guaranteed to disentangle state and noise that is faithful to the underlying dynamics. Empirical evidence from extensive benchmark control tasks demonstrates the superiority of our approach over existing counterparts in effectively disentangling state belief from noise. |
| title | Rethinking State Disentanglement in Causal Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2408.13498 |