An Analysis of Quantile Temporal-Difference Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911879832535040 |
|---|---|
| author | Rowland, Mark Munos, Rémi Azar, Mohammad Gheshlaghi Tang, Yunhao Ostrovski, Georg Harutyunyan, Anna Tuyls, Karl Bellemare, Marc G. Dabney, Will |
| author_facet | Rowland, Mark Munos, Rémi Azar, Mohammad Gheshlaghi Tang, Yunhao Ostrovski, Georg Harutyunyan, Anna Tuyls, Karl Bellemare, Marc G. Dabney, Will |
| contents | We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2301_04462 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | An Analysis of Quantile Temporal-Difference Learning Rowland, Mark Munos, Rémi Azar, Mohammad Gheshlaghi Tang, Yunhao Ostrovski, Georg Harutyunyan, Anna Tuyls, Karl Bellemare, Marc G. Dabney, Will Machine Learning We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis. |
| title | An Analysis of Quantile Temporal-Difference Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2301.04462 |