ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909831845117952 |
|---|---|
| author | Kim, Taewhan Bae, Hojin Li, Zeming Li, Xiaoqi Ponomarenko, Iaroslav Wu, Ruihai Dong, Hao |
| author_facet | Kim, Taewhan Bae, Hojin Li, Zeming Li, Xiaoqi Ponomarenko, Iaroslav Wu, Ruihai Dong, Hao |
| contents | Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10050 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? Kim, Taewhan Bae, Hojin Li, Zeming Li, Xiaoqi Ponomarenko, Iaroslav Wu, Ruihai Dong, Hao Robotics Computer Vision and Pattern Recognition Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems. |
| title | ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.10050 |