Leveraging Symmetry in RL-based Legged Locomotion Control
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912268884639744 |
|---|---|
| author | Su, Zhi Huang, Xiaoyu Ordoñez-Apraez, Daniel Li, Yunfei Li, Zhongyu Liao, Qiayuan Turrisi, Giulio Pontil, Massimiliano Semini, Claudio Wu, Yi Sreenath, Koushil |
| author_facet | Su, Zhi Huang, Xiaoyu Ordoñez-Apraez, Daniel Li, Yunfei Li, Zhongyu Liao, Qiayuan Turrisi, Giulio Pontil, Massimiliano Semini, Claudio Wu, Yi Sreenath, Koushil |
| contents | Model-free reinforcement learning is a promising approach for autonomously solving challenging robotics control problems, but faces exploration difficulty without information of the robot's kinematics and dynamics morphology. The under-exploration of multiple modalities with symmetric states leads to behaviors that are often unnatural and sub-optimal. This issue becomes particularly pronounced in the context of robotic systems with morphological symmetries, such as legged robots for which the resulting asymmetric and aperiodic behaviors compromise performance, robustness, and transferability to real hardware. To mitigate this challenge, we can leverage symmetry to guide and improve the exploration in policy learning via equivariance/invariance constraints. In this paper, we investigate the efficacy of two approaches to incorporate symmetry: modifying the network architectures to be strictly equivariant/invariant, and leveraging data augmentation to approximate equivariant/invariant actor-critics. We implement the methods on challenging loco-manipulation and bipedal locomotion tasks and compare with an unconstrained baseline. We find that the strictly equivariant policy consistently outperforms other methods in sample efficiency and task performance in simulation. In addition, symmetry-incorporated approaches exhibit better gait quality, higher robustness and can be deployed zero-shot in real-world experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leveraging Symmetry in RL-based Legged Locomotion Control Su, Zhi Huang, Xiaoyu Ordoñez-Apraez, Daniel Li, Yunfei Li, Zhongyu Liao, Qiayuan Turrisi, Giulio Pontil, Massimiliano Semini, Claudio Wu, Yi Sreenath, Koushil Robotics Model-free reinforcement learning is a promising approach for autonomously solving challenging robotics control problems, but faces exploration difficulty without information of the robot's kinematics and dynamics morphology. The under-exploration of multiple modalities with symmetric states leads to behaviors that are often unnatural and sub-optimal. This issue becomes particularly pronounced in the context of robotic systems with morphological symmetries, such as legged robots for which the resulting asymmetric and aperiodic behaviors compromise performance, robustness, and transferability to real hardware. To mitigate this challenge, we can leverage symmetry to guide and improve the exploration in policy learning via equivariance/invariance constraints. In this paper, we investigate the efficacy of two approaches to incorporate symmetry: modifying the network architectures to be strictly equivariant/invariant, and leveraging data augmentation to approximate equivariant/invariant actor-critics. We implement the methods on challenging loco-manipulation and bipedal locomotion tasks and compare with an unconstrained baseline. We find that the strictly equivariant policy consistently outperforms other methods in sample efficiency and task performance in simulation. In addition, symmetry-incorporated approaches exhibit better gait quality, higher robustness and can be deployed zero-shot in real-world experiments. |
| title | Leveraging Symmetry in RL-based Legged Locomotion Control |
| topic | Robotics |
| url | https://arxiv.org/abs/2403.17320 |