Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.19732 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929608897593344 |
|---|---|
| author | Bochem, Severin Gonzalez-Sanchez, Eduardo Bicker, Yves Fadini, Gabriele |
| author_facet | Bochem, Severin Gonzalez-Sanchez, Eduardo Bicker, Yves Fadini, Gabriele |
| contents | Reinforcement learning often requires extensive training data. Simulation-to-real transfer offers a promising approach to address this challenge in robotics. While differentiable simulators offer improved sample efficiency through exact gradients, they can be unstable in contact-rich environments and may lead to poor generalization. This paper introduces a novel approach integrating sharpness-aware optimization into gradient-based reinforcement learning algorithms. Our simulation results demonstrate that our method, tested on contact-rich environments, significantly enhances policy robustness to environmental variations and action perturbations while maintaining the sample efficiency of first-order methods. Specifically, our approach improves action noise tolerance compared to standard first-order methods and achieves generalization comparable to zeroth-order methods. This improvement stems from finding flatter minima in the loss landscape, associated with better generalization. Our work offers a promising solution to balance efficient learning and robust sim-to-real transfer in robotics, potentially bridging the gap between simulation and real-world performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_19732 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learning Bochem, Severin Gonzalez-Sanchez, Eduardo Bicker, Yves Fadini, Gabriele Robotics Artificial Intelligence Machine Learning Reinforcement learning often requires extensive training data. Simulation-to-real transfer offers a promising approach to address this challenge in robotics. While differentiable simulators offer improved sample efficiency through exact gradients, they can be unstable in contact-rich environments and may lead to poor generalization. This paper introduces a novel approach integrating sharpness-aware optimization into gradient-based reinforcement learning algorithms. Our simulation results demonstrate that our method, tested on contact-rich environments, significantly enhances policy robustness to environmental variations and action perturbations while maintaining the sample efficiency of first-order methods. Specifically, our approach improves action noise tolerance compared to standard first-order methods and achieves generalization comparable to zeroth-order methods. This improvement stems from finding flatter minima in the loss landscape, associated with better generalization. Our work offers a promising solution to balance efficient learning and robust sim-to-real transfer in robotics, potentially bridging the gap between simulation and real-world performance. |
| title | Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learning |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.19732 |