SAPG: Split and Aggregate Policy Gradients
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929440820297728 |
|---|---|
| author | Singla, Jayesh Agarwal, Ananye Pathak, Deepak |
| author_facet | Singla, Jayesh Agarwal, Ananye Pathak, Deepak |
| contents | Despite extreme sample inefficiency, on-policy reinforcement learning, aka policy gradients, has become a fundamental tool in decision-making problems. With the recent advances in GPU-driven simulation, the ability to collect large amounts of data for RL training has scaled exponentially. However, we show that current RL methods, e.g. PPO, fail to ingest the benefit of parallelized environments beyond a certain point and their performance saturates. To address this, we propose a new on-policy RL algorithm that can effectively leverage large-scale environments by splitting them into chunks and fusing them back together via importance sampling. Our algorithm, termed SAPG, shows significantly higher performance across a variety of challenging environments where vanilla PPO and other strong baselines fail to achieve high performance. Website at https://sapg-rl.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20230 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SAPG: Split and Aggregate Policy Gradients Singla, Jayesh Agarwal, Ananye Pathak, Deepak Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics Systems and Control Despite extreme sample inefficiency, on-policy reinforcement learning, aka policy gradients, has become a fundamental tool in decision-making problems. With the recent advances in GPU-driven simulation, the ability to collect large amounts of data for RL training has scaled exponentially. However, we show that current RL methods, e.g. PPO, fail to ingest the benefit of parallelized environments beyond a certain point and their performance saturates. To address this, we propose a new on-policy RL algorithm that can effectively leverage large-scale environments by splitting them into chunks and fusing them back together via importance sampling. Our algorithm, termed SAPG, shows significantly higher performance across a variety of challenging environments where vanilla PPO and other strong baselines fail to achieve high performance. Website at https://sapg-rl.github.io/ |
| title | SAPG: Split and Aggregate Policy Gradients |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics Systems and Control |
| url | https://arxiv.org/abs/2407.20230 |