Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.07775 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929418792861696 |
|---|---|
| author | Chiang, Hao-Tien Lewis Xu, Zhuo Fu, Zipeng Jacob, Mithun George Zhang, Tingnan Lee, Tsang-Wei Edward Yu, Wenhao Schenck, Connor Rendleman, David Shah, Dhruv Xia, Fei Hsu, Jasmine Hoech, Jonathan Florence, Pete Kirmani, Sean Singh, Sumeet Sindhwani, Vikas Parada, Carolina Finn, Chelsea Xu, Peng Levine, Sergey Tan, Jie |
| author_facet | Chiang, Hao-Tien Lewis Xu, Zhuo Fu, Zipeng Jacob, Mithun George Zhang, Tingnan Lee, Tsang-Wei Edward Yu, Wenhao Schenck, Connor Rendleman, David Shah, Dhruv Xia, Fei Hsu, Jasmine Hoech, Jonathan Florence, Pete Kirmani, Sean Singh, Sumeet Sindhwani, Vikas Parada, Carolina Finn, Chelsea Xu, Peng Levine, Sergey Tan, Jie |
| contents | An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07775 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs Chiang, Hao-Tien Lewis Xu, Zhuo Fu, Zipeng Jacob, Mithun George Zhang, Tingnan Lee, Tsang-Wei Edward Yu, Wenhao Schenck, Connor Rendleman, David Shah, Dhruv Xia, Fei Hsu, Jasmine Hoech, Jonathan Florence, Pete Kirmani, Sean Singh, Sumeet Sindhwani, Vikas Parada, Carolina Finn, Chelsea Xu, Peng Levine, Sergey Tan, Jie Robotics Artificial Intelligence An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s |
| title | Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2407.07775 |