Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909010762924032 |
|---|---|
| author | Zhang, Ruoning Wang, Siying Chen, Wenyu Zhou, Yang Zhao, Zhitong Zhang, Zixuan Zhang, Ruijie Albrecht, Stefano V. |
| author_facet | Zhang, Ruoning Wang, Siying Chen, Wenyu Zhou, Yang Zhao, Zhitong Zhang, Zixuan Zhang, Ruijie Albrecht, Stefano V. |
| contents | The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, conventional methods based on CTDE can suffer from value underestimation and converge to suboptimal solutions. While such underestimation is typically attributed to the representational limitations of monotonic structures, we provide a novel perspective by demonstrating that the insufficient sampling of optimal joint actions during exploration is also a critical factor. To address this problem, we propose Optimistic $ε$-Greedy Exploration. Our method introduces optimistic action-value networks that serve as decoupled exploration indicators, which we theoretically prove to converge in probability to the maximum achievable returns. By sampling actions from these distributions with a probability of $ε$, we effectively increase the selection frequency of high-return joint actions. Experimental results in various environments reveal that our strategy effectively prevents the algorithm from falling into suboptimal solutions and significantly improves final returns, win rates, and convergence speeds compared to other enhanced algorithms. Our code has been open-sourced at https://github.com/qxqxtxdy/OptimisticExploration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03506 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning Zhang, Ruoning Wang, Siying Chen, Wenyu Zhou, Yang Zhao, Zhitong Zhang, Zixuan Zhang, Ruijie Albrecht, Stefano V. Multiagent Systems Machine Learning The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, conventional methods based on CTDE can suffer from value underestimation and converge to suboptimal solutions. While such underestimation is typically attributed to the representational limitations of monotonic structures, we provide a novel perspective by demonstrating that the insufficient sampling of optimal joint actions during exploration is also a critical factor. To address this problem, we propose Optimistic $ε$-Greedy Exploration. Our method introduces optimistic action-value networks that serve as decoupled exploration indicators, which we theoretically prove to converge in probability to the maximum achievable returns. By sampling actions from these distributions with a probability of $ε$, we effectively increase the selection frequency of high-return joint actions. Experimental results in various environments reveal that our strategy effectively prevents the algorithm from falling into suboptimal solutions and significantly improves final returns, win rates, and convergence speeds compared to other enhanced algorithms. Our code has been open-sourced at https://github.com/qxqxtxdy/OptimisticExploration. |
| title | Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning |
| topic | Multiagent Systems Machine Learning |
| url | https://arxiv.org/abs/2502.03506 |