Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.15381 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910596237099008 |
|---|---|
| author | Marchesini, Enrico Baisero, Andrea Bhati, Rupali Amato, Christopher |
| author_facet | Marchesini, Enrico Baisero, Andrea Bhati, Rupali Amato, Christopher |
| contents | Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performance. For example, the theory in prior work uses stateless (i.e., history) functions, while the practical implementations use state information -- making the motivating theory a mismatch for the implementation. Also, methods have built off of previous approaches, inheriting their architectures without exploring other, potentially better ones. To address these concerns, we formally analyze the theory of using the state instead of the history in current methods -- reconnecting theory and practice. We then introduce DuelMIX, a factorization algorithm that learns distinct per-agent utility estimators to improve performance and achieve full expressiveness. Experiments on StarCraft II micromanagement and Box Pushing tasks demonstrate the benefits of our intuitions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15381 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Stateful Value Factorization in Multi-Agent Reinforcement Learning Marchesini, Enrico Baisero, Andrea Bhati, Rupali Amato, Christopher Artificial Intelligence Value factorization is a popular paradigm for designing scalable multi-agent reinforcement learning algorithms. However, current factorization methods make choices without full justification that may limit their performance. For example, the theory in prior work uses stateless (i.e., history) functions, while the practical implementations use state information -- making the motivating theory a mismatch for the implementation. Also, methods have built off of previous approaches, inheriting their architectures without exploring other, potentially better ones. To address these concerns, we formally analyze the theory of using the state instead of the history in current methods -- reconnecting theory and practice. We then introduce DuelMIX, a factorization algorithm that learns distinct per-agent utility estimators to improve performance and achieve full expressiveness. Experiments on StarCraft II micromanagement and Box Pushing tasks demonstrate the benefits of our intuitions. |
| title | On Stateful Value Factorization in Multi-Agent Reinforcement Learning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2408.15381 |