Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.15544 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911412130938880 |
|---|---|
| author | Castanyer, Roger Creus Obando-Ceron, Johan Li, Lu Bacon, Pierre-Luc Berseth, Glen Courville, Aaron Castro, Pablo Samuel |
| author_facet | Castanyer, Roger Creus Obando-Ceron, Johan Li, Lu Bacon, Pierre-Luc Berseth, Glen Courville, Aaron Castro, Pablo Samuel |
| contents | Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure mode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct a series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of scale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15544 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning Castanyer, Roger Creus Obando-Ceron, Johan Li, Lu Bacon, Pierre-Luc Berseth, Glen Courville, Aaron Castro, Pablo Samuel Machine Learning Scaling deep reinforcement learning networks is challenging and often results in degraded performance, yet the root causes of this failure mode remain poorly understood. Several recent works have proposed mechanisms to address this, but they are often complex and fail to highlight the causes underlying this difficulty. In this work, we conduct a series of empirical analyses which suggest that the combination of non-stationarity with gradient pathologies, due to suboptimal architectural choices, underlie the challenges of scale. We propose a series of direct interventions that stabilize gradient flow, enabling robust performance across a range of network depths and widths. Our interventions are simple to implement and compatible with well-established algorithms, and result in an effective mechanism that enables strong performance even at large scales. We validate our findings on a variety of agents and suites of environments. |
| title | Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.15544 |