Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913392965451776 |
|---|---|
| author | Li, Simin Guo, Jun Xiu, Jingqiao Xu, Ruixiao Yu, Xin Wang, Jiakai Liu, Aishan Yang, Yaodong Liu, Xianglong |
| author_facet | Li, Simin Guo, Jun Xiu, Jingqiao Xu, Ruixiao Yu, Xin Wang, Jiakai Liu, Aishan Yang, Yaodong Liu, Xianglong |
| contents | In this study, we explore the robustness of cooperative multi-agent reinforcement learning (c-MARL) against Byzantine failures, where any agent can enact arbitrary, worst-case actions due to malfunction or adversarial attack. To address the uncertainty that any agent can be adversarial, we propose a Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP) framework, which views Byzantine adversaries as nature-dictated types, represented by a separate transition. This allows agents to learn policies grounded on their posterior beliefs about the type of other agents, fostering collaboration with identified allies and minimizing vulnerability to adversarial manipulation. We define the optimal solution to the BARDec-POMDP as an ex post robust Bayesian Markov perfect equilibrium, which we proof to exist and weakly dominates the equilibrium of previous robust MARL approaches. To realize this equilibrium, we put forward a two-timescale actor-critic algorithm with almost sure convergence under specific conditions. Experimentation on matrix games, level-based foraging and StarCraft II indicate that, even under worst-case perturbations, our method successfully acquires intricate micromanagement skills and adaptively aligns with allies, demonstrating resilience against non-oblivious adversaries, random allies, observation-based attacks, and transfer-based attacks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_12872 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game Li, Simin Guo, Jun Xiu, Jingqiao Xu, Ruixiao Yu, Xin Wang, Jiakai Liu, Aishan Yang, Yaodong Liu, Xianglong Computer Science and Game Theory In this study, we explore the robustness of cooperative multi-agent reinforcement learning (c-MARL) against Byzantine failures, where any agent can enact arbitrary, worst-case actions due to malfunction or adversarial attack. To address the uncertainty that any agent can be adversarial, we propose a Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP) framework, which views Byzantine adversaries as nature-dictated types, represented by a separate transition. This allows agents to learn policies grounded on their posterior beliefs about the type of other agents, fostering collaboration with identified allies and minimizing vulnerability to adversarial manipulation. We define the optimal solution to the BARDec-POMDP as an ex post robust Bayesian Markov perfect equilibrium, which we proof to exist and weakly dominates the equilibrium of previous robust MARL approaches. To realize this equilibrium, we put forward a two-timescale actor-critic algorithm with almost sure convergence under specific conditions. Experimentation on matrix games, level-based foraging and StarCraft II indicate that, even under worst-case perturbations, our method successfully acquires intricate micromanagement skills and adaptively aligns with allies, demonstrating resilience against non-oblivious adversaries, random allies, observation-based attacks, and transfer-based attacks. |
| title | Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game |
| topic | Computer Science and Game Theory |
| url | https://arxiv.org/abs/2305.12872 |