CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929596932292608 |
|---|---|
| author | Shafiullah, Nur Muhammad Mahi Paxton, Chris Pinto, Lerrel Chintala, Soumith Szlam, Arthur |
| author_facet | Shafiullah, Nur Muhammad Mahi Paxton, Chris Pinto, Lerrel Chintala, Soumith Szlam, Arthur |
| contents | We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_05663 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory Shafiullah, Nur Muhammad Mahi Paxton, Chris Pinto, Lerrel Chintala, Soumith Szlam, Arthur Robotics Computer Vision and Pattern Recognition We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields |
| title | CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2210.05663 |