ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911953457250304 |
|---|---|
| author | Qi, Zekun Dong, Runpei Zhang, Shaochen Geng, Haoran Han, Chunrui Ge, Zheng Yi, Li Ma, Kaisheng |
| author_facet | Qi, Zekun Dong, Runpei Zhang, Shaochen Geng, Haoran Han, Chunrui Ge, Zheng Yi, Li Ma, Kaisheng |
| contents | This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17766 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ShapeLLM: Universal 3D Object Understanding for Embodied Interaction Qi, Zekun Dong, Runpei Zhang, Shaochen Geng, Haoran Han, Chunrui Ge, Zheng Yi, Li Ma, Kaisheng Computer Vision and Pattern Recognition This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages. ShapeLLM is built upon an improved 3D encoder by extending ReCon to ReCon++ that benefits from multi-view image distillation for enhanced geometry understanding. By utilizing ReCon++ as the 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructed instruction-following data and tested on our newly human-curated benchmark, 3D MM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3D geometry understanding and language-unified 3D interaction tasks, such as embodied visual grounding. Project page: https://qizekun.github.io/shapellm/ |
| title | ShapeLLM: Universal 3D Object Understanding for Embodied Interaction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.17766 |