ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915002105987072 |
|---|---|
| author | Ji, Tianying Liang, Yongyuan Zeng, Yan Luo, Yu Xu, Guowei Guo, Jiawei Zheng, Ruijie Huang, Furong Sun, Fuchun Xu, Huazhe |
| author_facet | Ji, Tianying Liang, Yongyuan Zeng, Yan Luo, Yu Xu, Guowei Guo, Jiawei Zheng, Ruijie Huang, Furong Sun, Fuchun Xu, Huazhe |
| contents | The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms. Leveraging this insight, we explore the causal relationship between different action dimensions and rewards to evaluate the significance of various primitive behaviors during training. We introduce a causality-aware entropy term that effectively identifies and prioritizes actions with high potential impacts for efficient exploration. Furthermore, to prevent excessive focus on specific primitive behaviors, we analyze the gradient dormancy phenomenon and introduce a dormancy-guided reset mechanism to further enhance the efficacy of our method. Our proposed algorithm, ACE: Off-policy Actor-critic with Causality-aware Entropy regularization, demonstrates a substantial performance advantage across 29 diverse continuous control tasks spanning 7 domains compared to model-free RL baselines, which underscores the effectiveness, versatility, and efficient sample efficiency of our approach. Benchmark results and videos are available at https://ace-rl.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14528 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization Ji, Tianying Liang, Yongyuan Zeng, Yan Luo, Yu Xu, Guowei Guo, Jiawei Zheng, Ruijie Huang, Furong Sun, Fuchun Xu, Huazhe Machine Learning Artificial Intelligence I.2 The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms. Leveraging this insight, we explore the causal relationship between different action dimensions and rewards to evaluate the significance of various primitive behaviors during training. We introduce a causality-aware entropy term that effectively identifies and prioritizes actions with high potential impacts for efficient exploration. Furthermore, to prevent excessive focus on specific primitive behaviors, we analyze the gradient dormancy phenomenon and introduce a dormancy-guided reset mechanism to further enhance the efficacy of our method. Our proposed algorithm, ACE: Off-policy Actor-critic with Causality-aware Entropy regularization, demonstrates a substantial performance advantage across 29 diverse continuous control tasks spanning 7 domains compared to model-free RL baselines, which underscores the effectiveness, versatility, and efficient sample efficiency of our approach. Benchmark results and videos are available at https://ace-rl.github.io/. |
| title | ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization |
| topic | Machine Learning Artificial Intelligence I.2 |
| url | https://arxiv.org/abs/2402.14528 |