Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929195397939200 |
|---|---|
| author | Shen, William Yang, Ge Yu, Alan Wong, Jansen Kaelbling, Leslie Pack Isola, Phillip |
| author_facet | Shen, William Yang, Ge Yu, Alan Wong, Jansen Kaelbling, Leslie Pack Isola, Phillip |
| contents | Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_07931 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation Shen, William Yang, Ge Yu, Alan Wong, Jansen Kaelbling, Leslie Pack Isola, Phillip Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. |
| title | Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics |
| url | https://arxiv.org/abs/2308.07931 |