A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2021
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911810938994688 |
|---|---|
| author | Feng, Xidong Liu, Bo Ren, Jie Mai, Luo Zhu, Rui Zhang, Haifeng Wang, Jun Yang, Yaodong |
| author_facet | Feng, Xidong Liu, Bo Ren, Jie Mai, Luo Zhu, Rui Zhang, Haifeng Wang, Jun Yang, Yaodong |
| contents | Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of $\mathcal{O}\big(Kα^{K}\hatσ_{\text{In}}|τ|^{-0.5}\big)$ \emph{w.r.t.} inner-loop update step $K$, learning rate $α$, estimate variance $\hatσ^{2}_{\text{In}}$ and sample size $|τ|$, and 2) the multi-step Hessian estimation bias $\hatΔ_{H}$ due to the use of autodiff, which has a polynomial impact $\mathcal{O}\big((K-1)(\hatΔ_{H})^{K-1}\big)$ on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2112_15400 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning Feng, Xidong Liu, Bo Ren, Jie Mai, Luo Zhu, Rui Zhang, Haifeng Wang, Jun Yang, Yaodong Machine Learning Artificial Intelligence Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of $\mathcal{O}\big(Kα^{K}\hatσ_{\text{In}}|τ|^{-0.5}\big)$ \emph{w.r.t.} inner-loop update step $K$, learning rate $α$, estimate variance $\hatσ^{2}_{\text{In}}$ and sample size $|τ|$, and 2) the multi-step Hessian estimation bias $\hatΔ_{H}$ due to the use of autodiff, which has a polynomial impact $\mathcal{O}\big((K-1)(\hatΔ_{H})^{K-1}\big)$ on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general. |
| title | A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2112.15400 |