Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929699898261504 |
|---|---|
| author | Zhang, Jiazhao Wang, Kunyu Wang, Shaoan Li, Minghan Liu, Haoran Wei, Songlin Wang, Zhongyuan Zhang, Zhizheng Wang, He |
| author_facet | Zhang, Jiazhao Wang, Kunyu Wang, Shaoan Li, Minghan Liu, Haoran Wei, Songlin Wang, Zhongyuan Zhang, Zhizheng Wang, He |
| contents | A practical navigation agent must be capable of handling a wide range of interaction demands, such as following instructions, searching objects, answering questions, tracking people, and more. Existing models for embodied navigation fall short of serving as practical generalists in the real world, as they are often constrained by specific task configurations or pre-defined maps with discretized waypoints. In this work, we present Uni-NaVid, the first video-based vision-language-action (VLA) model designed to unify diverse embodied navigation tasks and enable seamless navigation for mixed long-horizon tasks in unseen real-world environments. Uni-NaVid achieves this by harmonizing the input and output data configurations for all commonly used embodied navigation tasks and thereby integrating all tasks in one model. For training Uni-NaVid, we collect 3.6 million navigation data samples in total from four essential navigation sub-tasks and foster synergy in learning across them. Extensive experiments on comprehensive navigation benchmarks clearly demonstrate the advantages of unification modeling in Uni-NaVid and show it achieves state-of-the-art performance. Additionally, real-world experiments confirm the model's effectiveness and efficiency, shedding light on its strong generalizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06224 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks Zhang, Jiazhao Wang, Kunyu Wang, Shaoan Li, Minghan Liu, Haoran Wei, Songlin Wang, Zhongyuan Zhang, Zhizheng Wang, He Robotics Computer Vision and Pattern Recognition A practical navigation agent must be capable of handling a wide range of interaction demands, such as following instructions, searching objects, answering questions, tracking people, and more. Existing models for embodied navigation fall short of serving as practical generalists in the real world, as they are often constrained by specific task configurations or pre-defined maps with discretized waypoints. In this work, we present Uni-NaVid, the first video-based vision-language-action (VLA) model designed to unify diverse embodied navigation tasks and enable seamless navigation for mixed long-horizon tasks in unseen real-world environments. Uni-NaVid achieves this by harmonizing the input and output data configurations for all commonly used embodied navigation tasks and thereby integrating all tasks in one model. For training Uni-NaVid, we collect 3.6 million navigation data samples in total from four essential navigation sub-tasks and foster synergy in learning across them. Extensive experiments on comprehensive navigation benchmarks clearly demonstrate the advantages of unification modeling in Uni-NaVid and show it achieves state-of-the-art performance. Additionally, real-world experiments confirm the model's effectiveness and efficiency, shedding light on its strong generalizability. |
| title | Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.06224 |