Learning Generalizable Feature Fields for Mobile Manipulation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912133333123072 |
|---|---|
| author | Qiu, Ri-Zhao Hu, Yafei Song, Yuchen Yang, Ge Fu, Yang Ye, Jianglong Mu, Jiteng Yang, Ruihan Atanasov, Nikolay Scherer, Sebastian Wang, Xiaolong |
| author_facet | Qiu, Ri-Zhao Hu, Yafei Song, Yuchen Yang, Ge Fu, Yang Ye, Jianglong Mu, Jiteng Yang, Ruihan Atanasov, Nikolay Scherer, Sebastian Wang, Xiaolong |
| contents | An open problem in mobile manipulation is how to represent objects and scenes in a unified manner so that robots can use both for navigation and manipulation. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherent at an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We quantitatively evaluate GeFF's ability for open-vocabulary object-/part-level manipulation and show that GeFF outperforms point-based baselines in runtime and storage-accuracy trade-offs, with qualitative examples of semantics-aware navigation and articulated object manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Generalizable Feature Fields for Mobile Manipulation Qiu, Ri-Zhao Hu, Yafei Song, Yuchen Yang, Ge Fu, Yang Ye, Jianglong Mu, Jiteng Yang, Ruihan Atanasov, Nikolay Scherer, Sebastian Wang, Xiaolong Robotics Computer Vision and Pattern Recognition Machine Learning An open problem in mobile manipulation is how to represent objects and scenes in a unified manner so that robots can use both for navigation and manipulation. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherent at an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We quantitatively evaluate GeFF's ability for open-vocabulary object-/part-level manipulation and show that GeFF outperforms point-based baselines in runtime and storage-accuracy trade-offs, with qualitative examples of semantics-aware navigation and articulated object manipulation. |
| title | Learning Generalizable Feature Fields for Mobile Manipulation |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2403.07563 |