Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.01979 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908349606395904 |
|---|---|
| author | Zhao, Chenran Shi, Dianxi Wang, Mengzhu Xia, Jianqiang Yang, Huanhuan Jin, Songchang Yang, Shaowu Qiu, Chunping |
| author_facet | Zhao, Chenran Shi, Dianxi Wang, Mengzhu Xia, Jianqiang Yang, Huanhuan Jin, Songchang Yang, Shaowu Qiu, Chunping |
| contents | Current Hierarchical Reinforcement Learning (HRL) algorithms excel in long-horizon sequential decision-making tasks but still face two challenges: delay effects and spurious correlations. To address them, we propose a causal HRL approach called D3HRL. First, D3HRL models delayed effects as causal relationships across different time spans and employs distributed causal discovery to learn these relationships. Second, it employs conditional independence testing to eliminate spurious correlations. Finally, D3HRL constructs and trains hierarchical policies based on the identified true causal relationships. These three steps are iteratively executed, gradually exploring the complete causal chain of the task. Experiments conducted in 2D-MineCraft and MiniGrid show that D3HRL demonstrates superior sensitivity to delay effects and accurately identifies causal relationships, leading to reliable decision-making in complex environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01979 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection Zhao, Chenran Shi, Dianxi Wang, Mengzhu Xia, Jianqiang Yang, Huanhuan Jin, Songchang Yang, Shaowu Qiu, Chunping Machine Learning Current Hierarchical Reinforcement Learning (HRL) algorithms excel in long-horizon sequential decision-making tasks but still face two challenges: delay effects and spurious correlations. To address them, we propose a causal HRL approach called D3HRL. First, D3HRL models delayed effects as causal relationships across different time spans and employs distributed causal discovery to learn these relationships. Second, it employs conditional independence testing to eliminate spurious correlations. Finally, D3HRL constructs and trains hierarchical policies based on the identified true causal relationships. These three steps are iteratively executed, gradually exploring the complete causal chain of the task. Experiments conducted in 2D-MineCraft and MiniGrid show that D3HRL demonstrates superior sensitivity to delay effects and accurately identifies causal relationships, leading to reliable decision-making in complex environments. |
| title | D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.01979 |