Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913854556995584 |
|---|---|
| author | Zhang, Zhen Chu, Xiangyu Tang, Yunxi Zhao, Lulu Huang, Jing Jiang, Zhongliang Au, K. W. Samuel |
| author_facet | Zhang, Zhen Chu, Xiangyu Tang, Yunxi Zhao, Lulu Huang, Jing Jiang, Zhongliang Au, K. W. Samuel |
| contents | Manipulating elasto-plastic objects remains a significant challenge due to severe self-occlusion, difficulties of representation, and complicated dynamics. This work proposes a novel framework for elasto-plastic object manipulation with a quasi-static assumption for motions, leveraging 3D occupancy to represent such objects, a learned dynamics model trained with 3D occupancy, and a learning-based predictive control algorithm to address these challenges effectively. We build a novel data collection platform to collect full spatial information and propose a pipeline for generating a 3D occupancy dataset. To infer the 3D occupancy during manipulation, an occupancy prediction network is trained with multiple RGB images supervised by the generated dataset. We design a deep neural network empowered by a 3D convolution neural network (CNN) and a graph neural network (GNN) to predict the complex deformation with the inferred 3D occupancy results. A learning-based predictive control algorithm is introduced to plan the robot actions, incorporating a novel shape-based action initialization module specifically designed to improve the planner efficiency. The proposed framework in this paper can successfully shape the elasto-plastic objects into a given goal shape and has been verified in various experiments both in simulation and the real world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16249 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control Zhang, Zhen Chu, Xiangyu Tang, Yunxi Zhao, Lulu Huang, Jing Jiang, Zhongliang Au, K. W. Samuel Robotics Artificial Intelligence Manipulating elasto-plastic objects remains a significant challenge due to severe self-occlusion, difficulties of representation, and complicated dynamics. This work proposes a novel framework for elasto-plastic object manipulation with a quasi-static assumption for motions, leveraging 3D occupancy to represent such objects, a learned dynamics model trained with 3D occupancy, and a learning-based predictive control algorithm to address these challenges effectively. We build a novel data collection platform to collect full spatial information and propose a pipeline for generating a 3D occupancy dataset. To infer the 3D occupancy during manipulation, an occupancy prediction network is trained with multiple RGB images supervised by the generated dataset. We design a deep neural network empowered by a 3D convolution neural network (CNN) and a graph neural network (GNN) to predict the complex deformation with the inferred 3D occupancy results. A learning-based predictive control algorithm is introduced to plan the robot actions, incorporating a novel shape-based action initialization module specifically designed to improve the planner efficiency. The proposed framework in this paper can successfully shape the elasto-plastic objects into a given goal shape and has been verified in various experiments both in simulation and the real world. |
| title | Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2505.16249 |