The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916777740468224 |
|---|---|
| author | Mayor, Walter Obando-Ceron, Johan Courville, Aaron Castro, Pablo Samuel |
| author_facet | Mayor, Walter Obando-Ceron, Johan Courville, Aaron Castro, Pablo Samuel |
| contents | The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03404 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks Mayor, Walter Obando-Ceron, Johan Courville, Aaron Castro, Pablo Samuel Machine Learning Artificial Intelligence The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance. |
| title | The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.03404 |