GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914098126520320 |
|---|---|
| author | Escontrela, Alejandro Kerr, Justin Allshire, Arthur Frey, Jonas Duan, Rocky Sferrazza, Carmelo Abbeel, Pieter |
| author_facet | Escontrela, Alejandro Kerr, Justin Allshire, Arthur Frey, Jonas Duan, Rocky Sferrazza, Carmelo Abbeel, Pieter |
| contents | We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15352 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GaussGym: An open-source real-to-sim framework for learning locomotion from pixels Escontrela, Alejandro Kerr, Justin Allshire, Arthur Frey, Jonas Duan, Rocky Sferrazza, Carmelo Abbeel, Pieter Robotics Artificial Intelligence Graphics We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/. |
| title | GaussGym: An open-source real-to-sim framework for learning locomotion from pixels |
| topic | Robotics Artificial Intelligence Graphics |
| url | https://arxiv.org/abs/2510.15352 |