Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2210.12282 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909059814260736 |
|---|---|
| author | Piche, Alexandre Thomas, Valentin Marino, Joseph Pardinas, Rafael Marconi, Gian Maria Pal, Christopher Khan, Mohammad Emtiyaz |
| author_facet | Piche, Alexandre Thomas, Valentin Marino, Joseph Pardinas, Rafael Marconi, Gian Maria Pal, Christopher Khan, Mohammad Emtiyaz |
| contents | Bootstrapping is behind much of the successes of Deep Reinforcement Learning. However, learning the value function via bootstrapping often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization is still misunderstood. In this work, we show that they act as an implicit regularizer. This regularizer has disadvantages such as being inflexible and non convex. To overcome these issues, we propose an explicit Functional Regularization that is a convex regularizer in function space and can easily be tuned. We analyze the convergence of our method theoretically and empirically demonstrate that replacing Target Networks with the more theoretically grounded Functional Regularization approach leads to better sample efficiency and performance improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_12282 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Bridging the Gap Between Target Networks and Functional Regularization Piche, Alexandre Thomas, Valentin Marino, Joseph Pardinas, Rafael Marconi, Gian Maria Pal, Christopher Khan, Mohammad Emtiyaz Machine Learning Bootstrapping is behind much of the successes of Deep Reinforcement Learning. However, learning the value function via bootstrapping often leads to unstable training due to fast-changing target values. Target Networks are employed to stabilize training by using an additional set of lagging parameters to estimate the target values. Despite the popularity of Target Networks, their effect on the optimization is still misunderstood. In this work, we show that they act as an implicit regularizer. This regularizer has disadvantages such as being inflexible and non convex. To overcome these issues, we propose an explicit Functional Regularization that is a convex regularizer in function space and can easily be tuned. We analyze the convergence of our method theoretically and empirically demonstrate that replacing Target Networks with the more theoretically grounded Functional Regularization approach leads to better sample efficiency and performance improvements. |
| title | Bridging the Gap Between Target Networks and Functional Regularization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2210.12282 |