Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914821522325504 |
|---|---|
| author | Angiuli, Andrea Fouque, Jean-Pierre Laurière, Mathieu Zhang, Mengrui |
| author_facet | Angiuli, Andrea Fouque, Jean-Pierre Laurière, Mathieu Zhang, Mengrui |
| contents | Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agents. Our analysis uses a Q-table for finite state and action spaces updated at each discrete time-step over an infinite horizon. In [Angiuli et al., 2023], we proved convergence of two-timescale algorithms for MFG and MFC separately highlighting the need to follow multiple population distributions in the MFC case. Here, we integrate this feature for MFCG as well as three rates of update decreasing to zero in the proper ratios. Our technique of proof uses a generalization to three timescales of the two-timescale analysis in [Borkar, 1997]. We give a simple example satisfying the various hypothesis made in the proof of convergence and illustrating the performance of the algorithm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17017 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games Angiuli, Andrea Fouque, Jean-Pierre Laurière, Mathieu Zhang, Mengrui Optimization and Control Machine Learning Multiagent Systems Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agents. Our analysis uses a Q-table for finite state and action spaces updated at each discrete time-step over an infinite horizon. In [Angiuli et al., 2023], we proved convergence of two-timescale algorithms for MFG and MFC separately highlighting the need to follow multiple population distributions in the MFC case. Here, we integrate this feature for MFCG as well as three rates of update decreasing to zero in the proper ratios. Our technique of proof uses a generalization to three timescales of the two-timescale analysis in [Borkar, 1997]. We give a simple example satisfying the various hypothesis made in the proof of convergence and illustrating the performance of the algorithm. |
| title | Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games |
| topic | Optimization and Control Machine Learning Multiagent Systems |
| url | https://arxiv.org/abs/2405.17017 |