Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.01101 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913526756409344 |
|---|---|
| author | Zhan, Wenhao Fujimoto, Scott Zhu, Zheqing Lee, Jason D. Jiang, Daniel R. Efroni, Yonathan |
| author_facet | Zhan, Wenhao Fujimoto, Scott Zhu, Zheqing Lee, Jason D. Jiang, Daniel R. Efroni, Yonathan |
| contents | We study the problem of learning an approximate equilibrium in the offline multi-agent reinforcement learning (MARL) setting. We introduce a structural assumption -- the interaction rank -- and establish that functions with low interaction rank are significantly more robust to distribution shift compared to general ones. Leveraging this observation, we demonstrate that utilizing function classes with low interaction rank, when combined with regularization and no-regret learning, admits decentralized, computationally and statistically efficient learning in offline MARL. Our theoretical results are complemented by experiments that showcase the potential of critic architectures with low interaction rank in offline MARL, contrasting with commonly used single-agent value decomposition architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01101 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank Zhan, Wenhao Fujimoto, Scott Zhu, Zheqing Lee, Jason D. Jiang, Daniel R. Efroni, Yonathan Machine Learning We study the problem of learning an approximate equilibrium in the offline multi-agent reinforcement learning (MARL) setting. We introduce a structural assumption -- the interaction rank -- and establish that functions with low interaction rank are significantly more robust to distribution shift compared to general ones. Leveraging this observation, we demonstrate that utilizing function classes with low interaction rank, when combined with regularization and no-regret learning, admits decentralized, computationally and statistically efficient learning in offline MARL. Our theoretical results are complemented by experiments that showcase the potential of critic architectures with low interaction rank in offline MARL, contrasting with commonly used single-agent value decomposition architectures. |
| title | Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.01101 |