Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910678434971648 |
|---|---|
| author | Wang, Qianxu Deng, Congyue Lum, Tyler Ga Wei Chen, Yuanpei Yang, Yaodong Bohg, Jeannette Zhu, Yixin Guibas, Leonidas |
| author_facet | Wang, Qianxu Deng, Congyue Lum, Tyler Ga Wei Chen, Yuanpei Yang, Yaodong Bohg, Jeannette Zhu, Yixin Guibas, Leonidas |
| contents | One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. While distilled feature fields from large vision models have enabled semantic correspondences across 3D scenes, their features are point-based and restricted to object surfaces, limiting their capability of modeling complex semantic feature distributions for hand-object interactions. In this work, we propose the \textit{neural attention field} for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. Core to it is a transformer decoder that computes the cross-attention between any 3D query point with all the scene points, and provides the query point feature with an attention-based aggregation. We further propose a self-supervised framework for training the transformer decoder from only a few 3D pointclouds without hand demonstrations. Post-training, the attention field can be applied to novel scenes for semantics-aware dexterous grasping from one-shot demonstration. Experiments show that our method provides better optimization landscapes by encouraging the end-effector to focus on task-relevant scene regions, resulting in significant improvements in success rates on real robots compared with the feature-field-based methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23039 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping Wang, Qianxu Deng, Congyue Lum, Tyler Ga Wei Chen, Yuanpei Yang, Yaodong Bohg, Jeannette Zhu, Yixin Guibas, Leonidas Robotics Computer Vision and Pattern Recognition One-shot transfer of dexterous grasps to novel scenes with object and context variations has been a challenging problem. While distilled feature fields from large vision models have enabled semantic correspondences across 3D scenes, their features are point-based and restricted to object surfaces, limiting their capability of modeling complex semantic feature distributions for hand-object interactions. In this work, we propose the \textit{neural attention field} for representing semantic-aware dense feature fields in the 3D space by modeling inter-point relevance instead of individual point features. Core to it is a transformer decoder that computes the cross-attention between any 3D query point with all the scene points, and provides the query point feature with an attention-based aggregation. We further propose a self-supervised framework for training the transformer decoder from only a few 3D pointclouds without hand demonstrations. Post-training, the attention field can be applied to novel scenes for semantics-aware dexterous grasping from one-shot demonstration. Experiments show that our method provides better optimization landscapes by encouraging the end-effector to focus on task-relevant scene regions, resulting in significant improvements in success rates on real robots compared with the feature-field-based methods. |
| title | Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Grasping |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.23039 |