Offline Actor-Critic Reinforcement Learning Scales to Large Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909099023663104 |
|---|---|
| author | Springenberg, Jost Tobias Abdolmaleki, Abbas Zhang, Jingwei Groth, Oliver Bloesch, Michael Lampe, Thomas Brakel, Philemon Bechtle, Sarah Kapturowski, Steven Hafner, Roland Heess, Nicolas Riedmiller, Martin |
| author_facet | Springenberg, Jost Tobias Abdolmaleki, Abbas Zhang, Jingwei Groth, Oliver Bloesch, Michael Lampe, Thomas Brakel, Philemon Bechtle, Sarah Kapturowski, Steven Hafner, Roland Heess, Nicolas Riedmiller, Martin |
| contents | We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outperform strong, supervised, behavioral cloning baselines for multi-task training on a large dataset containing both sub-optimal and expert behavior on 132 continuous control tasks. We introduce a Perceiver-based actor-critic model and elucidate the key model features needed to make offline RL work with self- and cross-attention modules. Overall, we find that: i) simple offline actor critic algorithms are a natural choice for gradually moving away from the currently predominant paradigm of behavioral cloning, and ii) via offline RL it is possible to learn multi-task policies that master many domains simultaneously, including real robotics tasks, from sub-optimal demonstrations or self-generated data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05546 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Offline Actor-Critic Reinforcement Learning Scales to Large Models Springenberg, Jost Tobias Abdolmaleki, Abbas Zhang, Jingwei Groth, Oliver Bloesch, Michael Lampe, Thomas Brakel, Philemon Bechtle, Sarah Kapturowski, Steven Hafner, Roland Heess, Nicolas Riedmiller, Martin Machine Learning Artificial Intelligence Robotics We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outperform strong, supervised, behavioral cloning baselines for multi-task training on a large dataset containing both sub-optimal and expert behavior on 132 continuous control tasks. We introduce a Perceiver-based actor-critic model and elucidate the key model features needed to make offline RL work with self- and cross-attention modules. Overall, we find that: i) simple offline actor critic algorithms are a natural choice for gradually moving away from the currently predominant paradigm of behavioral cloning, and ii) via offline RL it is possible to learn multi-task policies that master many domains simultaneously, including real robotics tasks, from sub-optimal demonstrations or self-generated data. |
| title | Offline Actor-Critic Reinforcement Learning Scales to Large Models |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2402.05546 |