Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914748506832896 |
|---|---|
| author | Haarnoja, Tuomas Moran, Ben Lever, Guy Huang, Sandy H. Tirumala, Dhruva Humplik, Jan Wulfmeier, Markus Tunyasuvunakool, Saran Siegel, Noah Y. Hafner, Roland Bloesch, Michael Hartikainen, Kristian Byravan, Arunkumar Hasenclever, Leonard Tassa, Yuval Sadeghi, Fereshteh Batchelor, Nathan Casarini, Federico Saliceti, Stefano Game, Charles Sreendra, Neil Patel, Kushal Gwira, Marlon Huber, Andrea Hurley, Nicole Nori, Francesco Hadsell, Raia Heess, Nicolas |
| author_facet | Haarnoja, Tuomas Moran, Ben Lever, Guy Huang, Sandy H. Tirumala, Dhruva Humplik, Jan Wulfmeier, Markus Tunyasuvunakool, Saran Siegel, Noah Y. Hafner, Roland Bloesch, Michael Hartikainen, Kristian Byravan, Arunkumar Hasenclever, Leonard Tassa, Yuval Sadeghi, Fereshteh Batchelor, Nathan Casarini, Federico Saliceti, Stefano Game, Charles Sreendra, Neil Patel, Kushal Gwira, Marlon Huber, Andrea Hurley, Nicole Nori, Francesco Hadsell, Raia Heess, Nicolas |
| contents | We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_13653 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning Haarnoja, Tuomas Moran, Ben Lever, Guy Huang, Sandy H. Tirumala, Dhruva Humplik, Jan Wulfmeier, Markus Tunyasuvunakool, Saran Siegel, Noah Y. Hafner, Roland Bloesch, Michael Hartikainen, Kristian Byravan, Arunkumar Hasenclever, Leonard Tassa, Yuval Sadeghi, Fereshteh Batchelor, Nathan Casarini, Federico Saliceti, Stefano Game, Charles Sreendra, Neil Patel, Kushal Gwira, Marlon Huber, Andrea Hurley, Nicole Nori, Francesco Hadsell, Raia Heess, Nicolas Robotics Artificial Intelligence Machine Learning We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. The resulting agent exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and it transitions between them in a smooth, stable, and efficient manner. The agent's locomotion and tactical behavior adapts to specific game contexts in a way that would be impractical to manually design. The agent also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. Our agent was trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer. Although the robots are inherently fragile, basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way -- well beyond what is intuitively expected from the robot. Indeed, in experiments, they walked 181% faster, turned 302% faster, took 63% less time to get up, and kicked a ball 34% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives. |
| title | Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2304.13653 |