SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909281215840256 |
|---|---|
| author | Ehsani, Kiana Gupta, Tanmay Hendrix, Rose Salvador, Jordi Weihs, Luca Zeng, Kuo-Hao Singh, Kunal Pratap Kim, Yejin Han, Winson Herrasti, Alvaro Krishna, Ranjay Schwenk, Dustin VanderBilt, Eli Kembhavi, Aniruddha |
| author_facet | Ehsani, Kiana Gupta, Tanmay Hendrix, Rose Salvador, Jordi Weihs, Luca Zeng, Kuo-Hao Singh, Kunal Pratap Kim, Yejin Han, Winson Herrasti, Alvaro Krishna, Ranjay Schwenk, Dustin VanderBilt, Eli Kembhavi, Aniruddha |
| contents | Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely expensive. In this work, we show that imitating shortest-path planners in simulation produces agents that, given a language instruction, can proficiently navigate, explore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth map or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, SPOC architecture, powerful visual encoders paired with extensive image augmentation, and the dramatic scale and diversity of our training data: millions of frames of shortest-path-expert trajectories collected inside approximately 200,000 procedurally generated houses containing 40,000 unique 3D assets. Our models, data, training code, and newly proposed 10-task benchmarking suite CHORES are available in https://spoc-robot.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_02976 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World Ehsani, Kiana Gupta, Tanmay Hendrix, Rose Salvador, Jordi Weihs, Luca Zeng, Kuo-Hao Singh, Kunal Pratap Kim, Yejin Han, Winson Herrasti, Alvaro Krishna, Ranjay Schwenk, Dustin VanderBilt, Eli Kembhavi, Aniruddha Robotics Artificial Intelligence Computer Vision and Pattern Recognition Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely expensive. In this work, we show that imitating shortest-path planners in simulation produces agents that, given a language instruction, can proficiently navigate, explore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth map or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, SPOC architecture, powerful visual encoders paired with extensive image augmentation, and the dramatic scale and diversity of our training data: millions of frames of shortest-path-expert trajectories collected inside approximately 200,000 procedurally generated houses containing 40,000 unique 3D assets. Our models, data, training code, and newly proposed 10-task benchmarking suite CHORES are available in https://spoc-robot.github.io. |
| title | SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.02976 |