WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929420894208000 |
|---|---|
| author | Lin, Zhenxiang Peng, Xidong Cong, Peishan Zheng, Ge Sun, Yujin Hou, Yuenan Zhu, Xinge Yang, Sibei Ma, Yuexin |
| author_facet | Lin, Zhenxiang Peng, Xidong Cong, Peishan Zheng, Ge Sun, Yujin Hou, Yuenan Zhu, Xinge Yang, Sibei Ma, Yuexin |
| contents | We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, dubbed WildRefer, for this task by fully utilizing the rich appearance information in images, the position and geometric clues in point cloud as well as the semantic knowledge of language descriptions. Besides, we propose two novel datasets, i.e., STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios accompanied with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive experiments and ablation studies demonstrate that our method achieves state-of-the-art performance on the proposed benchmarks. The code is provided in https://github.com/4DVLab/WildRefer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_05645 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language Lin, Zhenxiang Peng, Xidong Cong, Peishan Zheng, Ge Sun, Yujin Hou, Yuenan Zhu, Xinge Yang, Sibei Ma, Yuexin Computer Vision and Pattern Recognition We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, dubbed WildRefer, for this task by fully utilizing the rich appearance information in images, the position and geometric clues in point cloud as well as the semantic knowledge of language descriptions. Besides, we propose two novel datasets, i.e., STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios accompanied with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive experiments and ablation studies demonstrate that our method achieves state-of-the-art performance on the proposed benchmarks. The code is provided in https://github.com/4DVLab/WildRefer. |
| title | WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2304.05645 |