Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2019
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/1902.05605 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909148252209152 |
|---|---|
| author | Bhatt, Aditya Palenicek, Daniel Belousov, Boris Argus, Max Amiranashvili, Artemij Brox, Thomas Peters, Jan |
| author_facet | Bhatt, Aditya Palenicek, Daniel Belousov, Boris Argus, Max Amiranashvili, Artemij Brox, Thomas Peters, Jan |
| contents | Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ: A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-art in sample efficiency while maintaining a low UTD ratio of 1. Notably, CrossQ does not rely on advanced bias-reduction schemes used in current methods. CrossQ's contributions are threefold: (1) it matches or surpasses current state-of-the-art methods in terms of sample efficiency, (2) it substantially reduces the computational cost compared to REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_1902_05605 |
| institution | arXiv |
| publishDate | 2019 |
| record_format | arxiv |
| spellingShingle | CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity Bhatt, Aditya Palenicek, Daniel Belousov, Boris Argus, Max Amiranashvili, Artemij Brox, Thomas Peters, Jan Machine Learning Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ: A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-art in sample efficiency while maintaining a low UTD ratio of 1. Notably, CrossQ does not rely on advanced bias-reduction schemes used in current methods. CrossQ's contributions are threefold: (1) it matches or surpasses current state-of-the-art methods in terms of sample efficiency, (2) it substantially reduces the computational cost compared to REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC. |
| title | CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity |
| topic | Machine Learning |
| url | https://arxiv.org/abs/1902.05605 |