Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917996527616000 |
|---|---|
| author | Cai, Ruichu Huang, Siyang Qiao, Jie Chen, Wei Zeng, Yan Zhang, Keli Sun, Fuchun Yu, Yang Hao, Zhifeng |
| author_facet | Cai, Ruichu Huang, Siyang Qiao, Jie Chen, Wei Zeng, Yan Zhang, Keli Sun, Fuchun Yu, Yang Hao, Zhifeng |
| contents | As a key component to intuitive cognition and reasoning solutions in human intelligence, causal knowledge provides great potential for reinforcement learning (RL) agents' interpretability towards decision-making by helping reduce the searching space. However, there is still a considerable gap in discovering and incorporating causality into RL, which hinders the rapid development of causal RL. In this paper, we consider explicitly modeling the generation process of states with the causal graphical model, based on which we augment the policy. We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventions for causal structure learning during exploration and using the learned causal structure for policy guidance during exploitation. Due to the lack of public benchmarks that allow direct intervention in the state space, we design the root cause localization task in our simulated fault alarm environment and then empirically show the effectiveness and robustness of the proposed method against state-of-the-art baselines. Theoretical analysis shows that our performance improvement attributes to the virtuous cycle of causal-guided policy learning and causal structure learning, which aligns with our experimental results. Codes are available at https://github.com/DMIRLAB-Group/FaultAlarm_RL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_04869 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy Cai, Ruichu Huang, Siyang Qiao, Jie Chen, Wei Zeng, Yan Zhang, Keli Sun, Fuchun Yu, Yang Hao, Zhifeng Machine Learning Artificial Intelligence As a key component to intuitive cognition and reasoning solutions in human intelligence, causal knowledge provides great potential for reinforcement learning (RL) agents' interpretability towards decision-making by helping reduce the searching space. However, there is still a considerable gap in discovering and incorporating causality into RL, which hinders the rapid development of causal RL. In this paper, we consider explicitly modeling the generation process of states with the causal graphical model, based on which we augment the policy. We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment. To optimize the derived objective, we propose a framework with theoretical performance guarantees that alternates between two steps: using interventions for causal structure learning during exploration and using the learned causal structure for policy guidance during exploitation. Due to the lack of public benchmarks that allow direct intervention in the state space, we design the root cause localization task in our simulated fault alarm environment and then empirically show the effectiveness and robustness of the proposed method against state-of-the-art baselines. Theoretical analysis shows that our performance improvement attributes to the virtuous cycle of causal-guided policy learning and causal structure learning, which aligns with our experimental results. Codes are available at https://github.com/DMIRLAB-Group/FaultAlarm_RL. |
| title | Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.04869 |