The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914137975554048 |
|---|---|
| author | Liu, Shuze Daniel Chen, Shuhang Zhang, Shangtong |
| author_facet | Liu, Shuze Daniel Chen, Shuhang Zhang, Shangtong |
| contents | Stochastic approximation is a class of algorithms that update a vector iteratively, incrementally, and stochastically, including, e.g., stochastic gradient descent and temporal difference learning. One fundamental challenge in analyzing a stochastic approximation algorithm is to establish its stability, i.e., to show that the stochastic vector iterates are bounded almost surely. In this paper, we extend the celebrated Borkar-Meyn theorem for stability from the Martingale difference noise setting to the Markovian noise setting, which greatly improves its applicability in reinforcement learning, especially in those off-policy reinforcement learning algorithms with linear function approximation and eligibility traces. Central to our analysis is the diminishing asymptotic rate of change of a few functions, which is implied by both a form of the strong law of large numbers and a form of the law of the iterated logarithm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_07844 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise Liu, Shuze Daniel Chen, Shuhang Zhang, Shangtong Machine Learning Artificial Intelligence Stochastic approximation is a class of algorithms that update a vector iteratively, incrementally, and stochastically, including, e.g., stochastic gradient descent and temporal difference learning. One fundamental challenge in analyzing a stochastic approximation algorithm is to establish its stability, i.e., to show that the stochastic vector iterates are bounded almost surely. In this paper, we extend the celebrated Borkar-Meyn theorem for stability from the Martingale difference noise setting to the Markovian noise setting, which greatly improves its applicability in reinforcement learning, especially in those off-policy reinforcement learning algorithms with linear function approximation and eligibility traces. Central to our analysis is the diminishing asymptotic rate of change of a few functions, which is implied by both a form of the strong law of large numbers and a form of the law of the iterated logarithm. |
| title | The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2401.07844 |