On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917793347141632 |
|---|---|
| author | Huang, Jiawei Yardim, Batuhan He, Niao |
| author_facet | Huang, Jiawei Yardim, Batuhan He, Niao |
| contents | In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an $ε$-optimal policy for MFC or an $ε$-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_11283 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation Huang, Jiawei Yardim, Batuhan He, Niao Machine Learning Artificial Intelligence In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an $ε$-optimal policy for MFC or an $ε$-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity. |
| title | On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2305.11283 |