RoboPearls: Editable Video Simulation for Robot Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911027511164928 |
|---|---|
| author | Tang, Tao Zhang, Likui Wen, Youpeng Zhang, Kaidong Bian, Jia-Wang zhou, xia Yan, Tianyi Zhan, Kun Jia, Peng Wu, Hefeng Lin, Liang Liang, Xiaodan |
| author_facet | Tang, Tao Zhang, Likui Wen, Youpeng Zhang, Kaidong Bian, Jia-Wang zhou, xia Yan, Tianyi Zhan, Kun Jia, Peng Wu, Hefeng Lin, Liang Liang, Xiaodan |
| contents | The development of generalist robot manipulation policies has seen significant progress, driven by large-scale demonstration data across diverse environments. However, the high cost and inefficiency of collecting real-world demonstrations hinder the scalability of data acquisition. While existing simulation platforms enable controlled environments for robotic learning, the challenge of bridging the sim-to-real gap remains. To address these challenges, we propose RoboPearls, an editable video simulation framework for robotic manipulation. Built on 3D Gaussian Splatting (3DGS), RoboPearls enables the construction of photo-realistic, view-consistent simulations from demonstration videos, and supports a wide range of simulation operators, including various object manipulations, powered by advanced modules like Incremental Semantic Distillation (ISD) and 3D regularized NNFM Loss (3D-NNFM). Moreover, by incorporating large language models (LLMs), RoboPearls automates the simulation production process in a user-friendly manner through flexible command interpretation and execution. Furthermore, RoboPearls employs a vision-language model (VLM) to analyze robotic learning issues to close the simulation loop for performance enhancement. To demonstrate the effectiveness of RoboPearls, we conduct extensive experiments on multiple datasets and scenes, including RLBench, COLOSSEUM, Ego4D, Open X-Embodiment, and a real-world robot, which demonstrate our satisfactory simulation performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22756 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoboPearls: Editable Video Simulation for Robot Manipulation Tang, Tao Zhang, Likui Wen, Youpeng Zhang, Kaidong Bian, Jia-Wang zhou, xia Yan, Tianyi Zhan, Kun Jia, Peng Wu, Hefeng Lin, Liang Liang, Xiaodan Computer Vision and Pattern Recognition Robotics The development of generalist robot manipulation policies has seen significant progress, driven by large-scale demonstration data across diverse environments. However, the high cost and inefficiency of collecting real-world demonstrations hinder the scalability of data acquisition. While existing simulation platforms enable controlled environments for robotic learning, the challenge of bridging the sim-to-real gap remains. To address these challenges, we propose RoboPearls, an editable video simulation framework for robotic manipulation. Built on 3D Gaussian Splatting (3DGS), RoboPearls enables the construction of photo-realistic, view-consistent simulations from demonstration videos, and supports a wide range of simulation operators, including various object manipulations, powered by advanced modules like Incremental Semantic Distillation (ISD) and 3D regularized NNFM Loss (3D-NNFM). Moreover, by incorporating large language models (LLMs), RoboPearls automates the simulation production process in a user-friendly manner through flexible command interpretation and execution. Furthermore, RoboPearls employs a vision-language model (VLM) to analyze robotic learning issues to close the simulation loop for performance enhancement. To demonstrate the effectiveness of RoboPearls, we conduct extensive experiments on multiple datasets and scenes, including RLBench, COLOSSEUM, Ego4D, Open X-Embodiment, and a real-world robot, which demonstrate our satisfactory simulation performance. |
| title | RoboPearls: Editable Video Simulation for Robot Manipulation |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2506.22756 |