Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.15536 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917046480011264 |
|---|---|
| author | Zook, Alex Sun, Fan-Yun Spjut, Josef Blukis, Valts Birchfield, Stan Tremblay, Jonathan |
| author_facet | Zook, Alex Sun, Fan-Yun Spjut, Josef Blukis, Valts Birchfield, Stan Tremblay, Jonathan |
| contents | We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training. Using vision-language models (VLMs), our pipeline operates in three stages: 1) scene comprehension with SAM2 for segmentation and object description, 2) matching objects with simulation-ready assets, and 3) generating appropriate tasks. We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code. Experiments demonstrate our system's effectiveness in object correspondence and task environment generation through our novel router mechanism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15536 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GRS: Generating Robotic Simulation Tasks from Real-World Images Zook, Alex Sun, Fan-Yun Spjut, Josef Blukis, Valts Birchfield, Stan Tremblay, Jonathan Robotics Artificial Intelligence We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training. Using vision-language models (VLMs), our pipeline operates in three stages: 1) scene comprehension with SAM2 for segmentation and object description, 2) matching objects with simulation-ready assets, and 3) generating appropriate tasks. We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code. Experiments demonstrate our system's effectiveness in object correspondence and task environment generation through our novel router mechanism. |
| title | GRS: Generating Robotic Simulation Tasks from Real-World Images |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2410.15536 |