Evaluating Real-World Robot Manipulation Policies in Simulation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914789495668736 |
|---|---|
| author | Li, Xuanlin Hsu, Kyle Gu, Jiayuan Pertsch, Karl Mees, Oier Walke, Homer Rich Fu, Chuyuan Lunawat, Ishikaa Sieh, Isabel Kirmani, Sean Levine, Sergey Wu, Jiajun Finn, Chelsea Su, Hao Vuong, Quan Xiao, Ted |
| author_facet | Li, Xuanlin Hsu, Kyle Gu, Jiayuan Pertsch, Karl Mees, Oier Walke, Homer Rich Fu, Chuyuan Lunawat, Ishikaa Sieh, Isabel Kirmani, Sean Levine, Sergey Wu, Jiajun Finn, Chelsea Su, Hao Vuong, Quan Xiao, Ted |
| contents | The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05941 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Real-World Robot Manipulation Policies in Simulation Li, Xuanlin Hsu, Kyle Gu, Jiayuan Pertsch, Karl Mees, Oier Walke, Homer Rich Fu, Chuyuan Lunawat, Ishikaa Sieh, Isabel Kirmani, Sean Levine, Sergey Wu, Jiajun Finn, Chelsea Su, Hao Vuong, Quan Xiao, Ted Robotics Computer Vision and Pattern Recognition Machine Learning The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. |
| title | Evaluating Real-World Robot Manipulation Policies in Simulation |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2405.05941 |