G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909655009067008 |
|---|---|
| author | Chen, Tianxing Mu, Yao Liang, Zhixuan Chen, Zanxin Peng, Shijia Chen, Qiangyu Xu, Mingkun Hu, Ruizhen Zhang, Hongyuan Li, Xuelong Luo, Ping |
| author_facet | Chen, Tianxing Mu, Yao Liang, Zhixuan Chen, Zanxin Peng, Shijia Chen, Qiangyu Xu, Mingkun Hu, Ruizhen Zhang, Hongyuan Li, Xuelong Luo, Ping |
| contents | Recent advances in imitation learning for 3D robotic manipulation have shown promising results with diffusion-based policies. However, achieving human-level dexterity requires seamless integration of geometric precision and semantic understanding. We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D semantic representation by leveraging foundation models. Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates. This integration enables complete semantic understanding even under occlusions while eliminating manual annotation requirements. By incorporating semantic flow into diffusion policies, we demonstrate significant improvements in both terminal-constrained manipulation and cross-object generalization. Extensive experiments across five simulation tasks show that G3Flow consistently outperforms existing approaches, achieving up to 68.3% and 50.1% average success rates on terminal-constrained manipulation and cross-object generalization tasks respectively. Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18369 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation Chen, Tianxing Mu, Yao Liang, Zhixuan Chen, Zanxin Peng, Shijia Chen, Qiangyu Xu, Mingkun Hu, Ruizhen Zhang, Hongyuan Li, Xuelong Luo, Ping Robotics Artificial Intelligence Computer Vision and Pattern Recognition Systems and Control Recent advances in imitation learning for 3D robotic manipulation have shown promising results with diffusion-based policies. However, achieving human-level dexterity requires seamless integration of geometric precision and semantic understanding. We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D semantic representation by leveraging foundation models. Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates. This integration enables complete semantic understanding even under occlusions while eliminating manual annotation requirements. By incorporating semantic flow into diffusion policies, we demonstrate significant improvements in both terminal-constrained manipulation and cross-object generalization. Extensive experiments across five simulation tasks show that G3Flow consistently outperforms existing approaches, achieving up to 68.3% and 50.1% average success rates on terminal-constrained manipulation and cross-object generalization tasks respectively. Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies. |
| title | G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Systems and Control |
| url | https://arxiv.org/abs/2411.18369 |