O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917991111720960 |
|---|---|
| author | Tie, Muer Wei, Julong Wang, Zhengjun Wu, Ke Yuan, Shansuai Zhang, Kaizhao Jia, Jie Zhao, Jieru Gan, Zhongxue Ding, Wenchao |
| author_facet | Tie, Muer Wei, Julong Wang, Zhengjun Wu, Ke Yuan, Shansuai Zhang, Kaizhao Jia, Jie Zhao, Jieru Gan, Zhongxue Ding, Wenchao |
| contents | Online construction of open-ended language scenes is crucial for robotic applications, where open-vocabulary interactive scene understanding is required. Recently, neural implicit representation has provided a promising direction for online interactive mapping. However, implementing open-vocabulary scene understanding capability into online neural implicit mapping still faces three challenges: lack of local scene updating ability, blurry spatial hierarchical semantic segmentation and difficulty in maintaining multi-view consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field, thus allowing for local updates during online training process. Additionally, we leverage a foundational model for image segmentation to extract language features on object-level entities, achieving clear segmentation boundaries and hierarchical semantic features. For the purpose of preserving consistency in 3D object properties across different viewpoints, we propose a spatial adaptive voxel adjustment mechanism and a multi-view weight selection method. Extensive experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes while enhancing accuracy, outperforming the previous SOTA method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06836 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation Tie, Muer Wei, Julong Wang, Zhengjun Wu, Ke Yuan, Shansuai Zhang, Kaizhao Jia, Jie Zhao, Jieru Gan, Zhongxue Ding, Wenchao Computer Vision and Pattern Recognition Online construction of open-ended language scenes is crucial for robotic applications, where open-vocabulary interactive scene understanding is required. Recently, neural implicit representation has provided a promising direction for online interactive mapping. However, implementing open-vocabulary scene understanding capability into online neural implicit mapping still faces three challenges: lack of local scene updating ability, blurry spatial hierarchical semantic segmentation and difficulty in maintaining multi-view consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field, thus allowing for local updates during online training process. Additionally, we leverage a foundational model for image segmentation to extract language features on object-level entities, achieving clear segmentation boundaries and hierarchical semantic features. For the purpose of preserving consistency in 3D object properties across different viewpoints, we propose a spatial adaptive voxel adjustment mechanism and a multi-view weight selection method. Extensive experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes while enhancing accuracy, outperforming the previous SOTA method. |
| title | O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.06836 |