3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911544681431040 |
|---|---|
| author | He, Yuxin Zhang, Ruihao Wu, Xianzu Zhang, Zhiyuan Ding, Cheng Nie, Qiang |
| author_facet | He, Yuxin Zhang, Ruihao Wu, Xianzu Zhang, Zhiyuan Ding, Cheng Nie, Qiang |
| contents | The incorporation of world modeling into manipulation policy learning has pushed the boundary of manipulation performance. However, existing efforts simply model the 2D visual dynamics, which is insufficient for robust manipulation when target tasks involve prominent depth-wise movement. To address this, we present a 3D dynamics-aware manipulation framework that seamlessly integrates 3D world modeling and policy learning. Three self-supervised learning tasks (current depth estimation, future RGB-D prediction, 3D flow prediction) are introduced within our framework, which complement each other and endow the policy model with 3D foresight. Extensive experiments on simulation and the real world show that 3D foresight can greatly boost the performance of manipulation policies without sacrificing inference speed. Code is available at https://github.com/Stardust-hyx/3D-Foresight. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10028 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | 3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight He, Yuxin Zhang, Ruihao Wu, Xianzu Zhang, Zhiyuan Ding, Cheng Nie, Qiang Computer Vision and Pattern Recognition Robotics The incorporation of world modeling into manipulation policy learning has pushed the boundary of manipulation performance. However, existing efforts simply model the 2D visual dynamics, which is insufficient for robust manipulation when target tasks involve prominent depth-wise movement. To address this, we present a 3D dynamics-aware manipulation framework that seamlessly integrates 3D world modeling and policy learning. Three self-supervised learning tasks (current depth estimation, future RGB-D prediction, 3D flow prediction) are introduced within our framework, which complement each other and endow the policy model with 3D foresight. Extensive experiments on simulation and the real world show that 3D foresight can greatly boost the performance of manipulation policies without sacrificing inference speed. Code is available at https://github.com/Stardust-hyx/3D-Foresight. |
| title | 3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2502.10028 |