Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916557680017408 |
|---|---|
| author | Zhang, Natalia Wang, Xinqi Cui, Qiwen Zhou, Runlong Kakade, Sham M. Du, Simon S. |
| author_facet | Zhang, Natalia Wang, Xinqi Cui, Qiwen Zhou, Runlong Kakade, Sham M. Du, Simon S. |
| contents | We initiate the study of Preference-Based Multi-Agent Reinforcement Learning (PbMARL), exploring both theoretical foundations and empirical validations. We define the task as identifying the Nash equilibrium from a preference-only offline dataset in general-sum games, a problem marked by the challenge of sparse feedback signals. Our theory establishes the upper complexity bounds for Nash Equilibrium in effective PbMARL, demonstrating that single-policy coverage is inadequate and highlighting the importance of unilateral dataset coverage. These theoretical insights are verified through comprehensive experiments. To enhance the practical performance, we further introduce two algorithmic techniques. (1) We propose a Mean Squared Error (MSE) regularization along the time axis to achieve a more uniform reward distribution and improve reward learning outcomes. (2) We propose an additional penalty based on the distribution of the dataset to incorporate pessimism, improving stability and effectiveness during training. Our findings underscore the multifaceted approach required for PbMARL, paving the way for effective preference-based multi-agent systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00717 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques Zhang, Natalia Wang, Xinqi Cui, Qiwen Zhou, Runlong Kakade, Sham M. Du, Simon S. Machine Learning Artificial Intelligence Computer Science and Game Theory Multiagent Systems We initiate the study of Preference-Based Multi-Agent Reinforcement Learning (PbMARL), exploring both theoretical foundations and empirical validations. We define the task as identifying the Nash equilibrium from a preference-only offline dataset in general-sum games, a problem marked by the challenge of sparse feedback signals. Our theory establishes the upper complexity bounds for Nash Equilibrium in effective PbMARL, demonstrating that single-policy coverage is inadequate and highlighting the importance of unilateral dataset coverage. These theoretical insights are verified through comprehensive experiments. To enhance the practical performance, we further introduce two algorithmic techniques. (1) We propose a Mean Squared Error (MSE) regularization along the time axis to achieve a more uniform reward distribution and improve reward learning outcomes. (2) We propose an additional penalty based on the distribution of the dataset to incorporate pessimism, improving stability and effectiveness during training. Our findings underscore the multifaceted approach required for PbMARL, paving the way for effective preference-based multi-agent systems. |
| title | Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques |
| topic | Machine Learning Artificial Intelligence Computer Science and Game Theory Multiagent Systems |
| url | https://arxiv.org/abs/2409.00717 |