DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909761646100480 |
|---|---|
| author | Wang, Zhaowei Zhang, Hongming Fang, Tianqing Tian, Ye Yang, Yue Ma, Kaixin Pan, Xiaoman Song, Yangqiu Yu, Dong |
| author_facet | Wang, Zhaowei Zhang, Hongming Fang, Tianqing Tian, Ye Yang, Yue Ma, Kaixin Pan, Xiaoman Song, Yangqiu Yu, Dong |
| contents | Large Vision-Language Models (LVLMs) have achieved significant progress in tasks like visual question answering and document understanding. However, their potential to comprehend embodied environments and navigate within them remains underexplored. In this work, we first study the challenge of open-vocabulary object navigation by introducing DivScene, a large-scale dataset with 4,614 houses across 81 scene types and 5,707 kinds of target objects. Our dataset provides a much greater diversity of target objects and scene types than existing datasets, enabling a comprehensive task evaluation. We evaluated various methods with LVLMs and LLMs on our dataset and found that current models still fall short of open-vocab object navigation ability. Then, we fine-tuned LVLMs to predict the next action with CoT explanations. We observe that LVLM's navigation ability can be improved substantially with only BFS-generated shortest paths without any human supervision, surpassing GPT-4o by over 20% in success rates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02730 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes Wang, Zhaowei Zhang, Hongming Fang, Tianqing Tian, Ye Yang, Yue Ma, Kaixin Pan, Xiaoman Song, Yangqiu Yu, Dong Computer Vision and Pattern Recognition Computation and Language Robotics Large Vision-Language Models (LVLMs) have achieved significant progress in tasks like visual question answering and document understanding. However, their potential to comprehend embodied environments and navigate within them remains underexplored. In this work, we first study the challenge of open-vocabulary object navigation by introducing DivScene, a large-scale dataset with 4,614 houses across 81 scene types and 5,707 kinds of target objects. Our dataset provides a much greater diversity of target objects and scene types than existing datasets, enabling a comprehensive task evaluation. We evaluated various methods with LVLMs and LLMs on our dataset and found that current models still fall short of open-vocab object navigation ability. Then, we fine-tuned LVLMs to predict the next action with CoT explanations. We observe that LVLM's navigation ability can be improved substantially with only BFS-generated shortest paths without any human supervision, surpassing GPT-4o by over 20% in success rates. |
| title | DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes |
| topic | Computer Vision and Pattern Recognition Computation and Language Robotics |
| url | https://arxiv.org/abs/2410.02730 |