SpatialBot: Precise Spatial Understanding with Vision Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909541540560896 |
|---|---|
| author | Cai, Wenxiao Ponomarenko, Iaroslav Yuan, Jianhao Li, Xiaoqi Yang, Wankou Dong, Hao Zhao, Bo |
| author_facet | Cai, Wenxiao Ponomarenko, Iaroslav Yuan, Jianhao Li, Xiaoqi Yang, Wankou Dong, Hao Zhao, Bo |
| contents | Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose SpatialBot for better spatial understanding by feeding both RGB and depth images. Additionally, we have constructed the SpatialQA dataset, which involves multi-level depth-related questions to train VLMs for depth understanding. Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities in spatial understanding at different levels. Extensive experiments on our spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks, demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The model, code and data are available at https://github.com/BAAI-DCAI/SpatialBot. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13642 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpatialBot: Precise Spatial Understanding with Vision Language Models Cai, Wenxiao Ponomarenko, Iaroslav Yuan, Jianhao Li, Xiaoqi Yang, Wankou Dong, Hao Zhao, Bo Computer Vision and Pattern Recognition Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose SpatialBot for better spatial understanding by feeding both RGB and depth images. Additionally, we have constructed the SpatialQA dataset, which involves multi-level depth-related questions to train VLMs for depth understanding. Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities in spatial understanding at different levels. Extensive experiments on our spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks, demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The model, code and data are available at https://github.com/BAAI-DCAI/SpatialBot. |
| title | SpatialBot: Precise Spatial Understanding with Vision Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.13642 |