Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917585182785536 |
|---|---|
| author | Ghaemi, Hafez Kebriaei, Hamed Moghaddam, Alireza Ramezani Ahamdabadi, Majid Nili |
| author_facet | Ghaemi, Hafez Kebriaei, Hamed Moghaddam, Alireza Ramezani Ahamdabadi, Majid Nili |
| contents | Classical multi-agent reinforcement learning (MARL) assumes risk neutrality and complete objectivity for agents. However, in settings where agents need to consider or model human economic or social preferences, a notion of risk must be incorporated into the RL optimization problem. This will be of greater importance in MARL where other human or non-human agents are involved, possibly with their own risk-sensitive policies. In this work, we consider risk-sensitive and non-cooperative MARL with cumulative prospect theory (CPT), a non-convex risk measure and a generalization of coherent measures of risk. CPT is capable of explaining loss aversion in humans and their tendency to overestimate/underestimate small/large probabilities. We propose a distributed sampling-based actor-critic (AC) algorithm with CPT risk for network aggregative Markov games (NAMGs), which we call Distributed Nested CPT-AC. Under a set of assumptions, we prove the convergence of the algorithm to a subjective notion of Markov perfect Nash equilibrium in NAMGs. The experimental results show that subjective CPT policies obtained by our algorithm can be different from the risk-neutral ones, and agents with a higher loss aversion are more inclined to socially isolate themselves in an NAMG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05906 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games Ghaemi, Hafez Kebriaei, Hamed Moghaddam, Alireza Ramezani Ahamdabadi, Majid Nili Machine Learning Artificial Intelligence Multiagent Systems I.2.6; I.2.11 Classical multi-agent reinforcement learning (MARL) assumes risk neutrality and complete objectivity for agents. However, in settings where agents need to consider or model human economic or social preferences, a notion of risk must be incorporated into the RL optimization problem. This will be of greater importance in MARL where other human or non-human agents are involved, possibly with their own risk-sensitive policies. In this work, we consider risk-sensitive and non-cooperative MARL with cumulative prospect theory (CPT), a non-convex risk measure and a generalization of coherent measures of risk. CPT is capable of explaining loss aversion in humans and their tendency to overestimate/underestimate small/large probabilities. We propose a distributed sampling-based actor-critic (AC) algorithm with CPT risk for network aggregative Markov games (NAMGs), which we call Distributed Nested CPT-AC. Under a set of assumptions, we prove the convergence of the algorithm to a subjective notion of Markov perfect Nash equilibrium in NAMGs. The experimental results show that subjective CPT policies obtained by our algorithm can be different from the risk-neutral ones, and agents with a higher loss aversion are more inclined to socially isolate themselves in an NAMG. |
| title | Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games |
| topic | Machine Learning Artificial Intelligence Multiagent Systems I.2.6; I.2.11 |
| url | https://arxiv.org/abs/2402.05906 |