FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstruction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913540693032960 |
|---|---|
| author | Fang, Irving Shi, Kairui He, Xujin Tan, Siqi Wang, Yifan Zhao, Hanwen Huang, Hung-Jui Yuan, Wenzhen Feng, Chen Zhang, Jing |
| author_facet | Fang, Irving Shi, Kairui He, Xujin Tan, Siqi Wang, Yifan Zhao, Hanwen Huang, Hung-Jui Yuan, Wenzhen Feng, Chen Zhang, Jing |
| contents | Humans effortlessly integrate common-sense knowledge with sensory input from vision and touch to understand their surroundings. Emulating this capability, we introduce FusionSense, a novel 3D reconstruction framework that enables robots to fuse priors from foundation models with highly sparse observations from vision and tactile sensors. FusionSense addresses three key challenges: (i) How can robots efficiently acquire robust global shape information about the surrounding scene and objects? (ii) How can robots strategically select touch points on the object using geometric and common-sense priors? (iii) How can partial observations such as tactile signals improve the overall representation of the object? Our framework employs 3D Gaussian Splatting as a core representation and incorporates a hierarchical optimization strategy involving global structure construction, object visual hull pruning and local geometric constraints. This advancement results in fast and robust perception in environments with traditionally challenging objects that are transparent, reflective, or dark, enabling more downstream manipulation or navigation tasks. Experiments on real-world data suggest that our framework outperforms previously state-of-the-art sparse-view methods. All code and data are open-sourced on the project website. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08282 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstruction Fang, Irving Shi, Kairui He, Xujin Tan, Siqi Wang, Yifan Zhao, Hanwen Huang, Hung-Jui Yuan, Wenzhen Feng, Chen Zhang, Jing Robotics Artificial Intelligence Computer Vision and Pattern Recognition Graphics I.4.5; I.4.8 Humans effortlessly integrate common-sense knowledge with sensory input from vision and touch to understand their surroundings. Emulating this capability, we introduce FusionSense, a novel 3D reconstruction framework that enables robots to fuse priors from foundation models with highly sparse observations from vision and tactile sensors. FusionSense addresses three key challenges: (i) How can robots efficiently acquire robust global shape information about the surrounding scene and objects? (ii) How can robots strategically select touch points on the object using geometric and common-sense priors? (iii) How can partial observations such as tactile signals improve the overall representation of the object? Our framework employs 3D Gaussian Splatting as a core representation and incorporates a hierarchical optimization strategy involving global structure construction, object visual hull pruning and local geometric constraints. This advancement results in fast and robust perception in environments with traditionally challenging objects that are transparent, reflective, or dark, enabling more downstream manipulation or navigation tasks. Experiments on real-world data suggest that our framework outperforms previously state-of-the-art sparse-view methods. All code and data are open-sourced on the project website. |
| title | FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstruction |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Graphics I.4.5; I.4.8 |
| url | https://arxiv.org/abs/2410.08282 |