CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910920024784896 |
|---|---|
| author | Fu, Yankai Feng, Qiuxuan Chen, Ning Zhou, Zichen Liu, Mengzhen Wu, Mingdong Chen, Tianxing Rong, Shanyu Liu, Jiaming Dong, Hao Zhang, Shanghang |
| author_facet | Fu, Yankai Feng, Qiuxuan Chen, Ning Zhou, Zichen Liu, Mengzhen Wu, Mingdong Chen, Tianxing Rong, Shanyu Liu, Jiaming Dong, Hao Zhang, Shanghang |
| contents | Achieving human-level dexterity in robots is a key objective in the field of robotic manipulation. Recent advancements in 3D-based imitation learning have shown promising results, providing an effective pathway to achieve this goal. However, obtaining high-quality 3D representations presents two key problems: (1) the quality of point clouds captured by a single-view camera is significantly affected by factors such as camera resolution, positioning, and occlusions caused by the dexterous hand; (2) the global point clouds lack crucial contact information and spatial correspondences, which are necessary for fine-grained dexterous manipulation tasks. To eliminate these limitations, we propose CordViP, a novel framework that constructs and learns correspondences by leveraging the robust 6D pose estimation of objects and robot proprioception. Specifically, we first introduce the interaction-aware point clouds, which establish correspondences between the object and the hand. These point clouds are then used for our pre-training policy, where we also incorporate object-centric contact maps and hand-arm coordination information, effectively capturing both spatial and temporal dynamics. Our method demonstrates exceptional dexterous manipulation capabilities, achieving state-of-the-art performance in six real-world tasks, surpassing other baselines by a large margin. Experimental results also highlight the superior generalization and robustness of CordViP to different objects, viewpoints, and scenarios. Code and videos are available on https://aureleopku.github.io/CordViP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08449 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World Fu, Yankai Feng, Qiuxuan Chen, Ning Zhou, Zichen Liu, Mengzhen Wu, Mingdong Chen, Tianxing Rong, Shanyu Liu, Jiaming Dong, Hao Zhang, Shanghang Robotics Artificial Intelligence Achieving human-level dexterity in robots is a key objective in the field of robotic manipulation. Recent advancements in 3D-based imitation learning have shown promising results, providing an effective pathway to achieve this goal. However, obtaining high-quality 3D representations presents two key problems: (1) the quality of point clouds captured by a single-view camera is significantly affected by factors such as camera resolution, positioning, and occlusions caused by the dexterous hand; (2) the global point clouds lack crucial contact information and spatial correspondences, which are necessary for fine-grained dexterous manipulation tasks. To eliminate these limitations, we propose CordViP, a novel framework that constructs and learns correspondences by leveraging the robust 6D pose estimation of objects and robot proprioception. Specifically, we first introduce the interaction-aware point clouds, which establish correspondences between the object and the hand. These point clouds are then used for our pre-training policy, where we also incorporate object-centric contact maps and hand-arm coordination information, effectively capturing both spatial and temporal dynamics. Our method demonstrates exceptional dexterous manipulation capabilities, achieving state-of-the-art performance in six real-world tasks, surpassing other baselines by a large margin. Experimental results also highlight the superior generalization and robustness of CordViP to different objects, viewpoints, and scenarios. Code and videos are available on https://aureleopku.github.io/CordViP. |
| title | CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2502.08449 |