QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910741276131328 |
|---|---|
| author | Mehan, Yash Gupta, Kumaraditya Jayanti, Rohit Govil, Anirudh Garg, Sourav Krishna, Madhava |
| author_facet | Mehan, Yash Gupta, Kumaraditya Jayanti, Rohit Govil, Anirudh Garg, Sourav Krishna, Madhava |
| contents | Robotic tasks such as planning and navigation require a hierarchical semantic understanding of a scene, which could include multiple floors and rooms. Current methods primarily focus on object segmentation for 3D scene understanding. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline to solve this problem. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding. Project Page: quest-maps.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06442 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding Mehan, Yash Gupta, Kumaraditya Jayanti, Rohit Govil, Anirudh Garg, Sourav Krishna, Madhava Computer Vision and Pattern Recognition Robotics Robotic tasks such as planning and navigation require a hierarchical semantic understanding of a scene, which could include multiple floors and rooms. Current methods primarily focus on object segmentation for 3D scene understanding. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline to solve this problem. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding. Project Page: quest-maps.github.io |
| title | QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2404.06442 |