RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917618758189056 |
|---|---|
| author | Shen, Siqi Ma, Chennan Li, Chao Liu, Weiquan Fu, Yongquan Mei, Songzhu Liu, Xinwang Wang, Cheng |
| author_facet | Shen, Siqi Ma, Chennan Li, Chao Liu, Weiquan Fu, Yongquan Mei, Songzhu Liu, Xinwang Wang, Cheng |
| contents | Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_01753 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization Shen, Siqi Ma, Chennan Li, Chao Liu, Weiquan Fu, Yongquan Mei, Songzhu Liu, Xinwang Wang, Cheng Multiagent Systems Artificial Intelligence Machine Learning Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ. |
| title | RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization |
| topic | Multiagent Systems Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2311.01753 |