Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929286357712896 |
|---|---|
| author | Yuan, Zhihao Ren, Jinke Feng, Chun-Mei Zhao, Hengshuang Cui, Shuguang Li, Zhen |
| author_facet | Yuan, Zhihao Ren, Jinke Feng, Chun-Mei Zhao, Hengshuang Cui, Shuguang Li, Zhen |
| contents | 3D Visual Grounding (3DVG) aims at localizing 3D object based on textual descriptions. Conventional supervised methods for 3DVG often necessitate extensive annotations and a predefined vocabulary, which can be restrictive. To address this issue, we propose a novel visual programming approach for zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language models (LLMs). Our approach begins with a unique dialog-based method, engaging with LLMs to establish a foundational understanding of zero-shot 3DVG. Building on this, we design a visual program that consists of three types of modules, i.e., view-independent, view-dependent, and functional modules. These modules, specifically tailored for 3D scenarios, work collaboratively to perform complex reasoning and inference. Furthermore, we develop an innovative language-object correlation module to extend the scope of existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot approach can outperform some supervised baselines, marking a significant stride towards effective 3DVG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_15383 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding Yuan, Zhihao Ren, Jinke Feng, Chun-Mei Zhao, Hengshuang Cui, Shuguang Li, Zhen Computer Vision and Pattern Recognition 3D Visual Grounding (3DVG) aims at localizing 3D object based on textual descriptions. Conventional supervised methods for 3DVG often necessitate extensive annotations and a predefined vocabulary, which can be restrictive. To address this issue, we propose a novel visual programming approach for zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language models (LLMs). Our approach begins with a unique dialog-based method, engaging with LLMs to establish a foundational understanding of zero-shot 3DVG. Building on this, we design a visual program that consists of three types of modules, i.e., view-independent, view-dependent, and functional modules. These modules, specifically tailored for 3D scenarios, work collaboratively to perform complex reasoning and inference. Furthermore, we develop an innovative language-object correlation module to extend the scope of existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot approach can outperform some supervised baselines, marking a significant stride towards effective 3DVG. |
| title | Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.15383 |