Identifying Policy Gradient Subspaces
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929280772997120 |
|---|---|
| author | Schneider, Jan Schumacher, Pierre Guist, Simon Chen, Le Häufle, Daniel Schölkopf, Bernhard Büchler, Dieter |
| author_facet | Schneider, Jan Schumacher, Pierre Guist, Simon Chen, Le Häufle, Daniel Schölkopf, Bernhard Büchler, Dieter |
| contents | Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06604 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Identifying Policy Gradient Subspaces Schneider, Jan Schumacher, Pierre Guist, Simon Chen, Le Häufle, Daniel Schölkopf, Bernhard Büchler, Dieter Machine Learning I.2.6 Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization. |
| title | Identifying Policy Gradient Subspaces |
| topic | Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2401.06604 |