Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.05951 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908544440205312 |
|---|---|
| author | van Oort, Tjeard Miller, Dimity Browne, Will N. Marticorena, Nicolas Haviland, Jesse Suenderhauf, Niko |
| author_facet | van Oort, Tjeard Miller, Dimity Browne, Will N. Marticorena, Nicolas Haviland, Jesse Suenderhauf, Niko |
| contents | Many robotic tasks require grasping objects at specific object parts instead of arbitrarily, a crucial capability for interactions beyond simple pick-and-place, such as human-robot interaction, handovers, or tool use. Prior work has focused either on generic grasp prediction or task-conditioned grasping, but not on directly targeting object parts in an open-vocabulary way. We propose AnyPart, a modular framework that unifies open-vocabulary object detection, part segmentation, and 6-DoF grasp prediction to enable robots to grasp user-specified parts of arbitrary objects based on natural language prompts. We evaluate 16 model combinations, and demonstrate that the best-performing combination achieves 60.8% grasp success in cluttered real-world scenes at 60 times faster inference than existing approaches. To support this study, we introduce a new dataset for part-based grasping and conduct a detailed failure analysis. Our core insight is that modularly combining existing foundation models unlocks surprisingly strong and efficient capabilities for open-vocabulary part-based grasping without requiring additional training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05951 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Open-Vocabulary Part-Based Grasping van Oort, Tjeard Miller, Dimity Browne, Will N. Marticorena, Nicolas Haviland, Jesse Suenderhauf, Niko Robotics Many robotic tasks require grasping objects at specific object parts instead of arbitrarily, a crucial capability for interactions beyond simple pick-and-place, such as human-robot interaction, handovers, or tool use. Prior work has focused either on generic grasp prediction or task-conditioned grasping, but not on directly targeting object parts in an open-vocabulary way. We propose AnyPart, a modular framework that unifies open-vocabulary object detection, part segmentation, and 6-DoF grasp prediction to enable robots to grasp user-specified parts of arbitrary objects based on natural language prompts. We evaluate 16 model combinations, and demonstrate that the best-performing combination achieves 60.8% grasp success in cluttered real-world scenes at 60 times faster inference than existing approaches. To support this study, we introduce a new dataset for part-based grasping and conduct a detailed failure analysis. Our core insight is that modularly combining existing foundation models unlocks surprisingly strong and efficient capabilities for open-vocabulary part-based grasping without requiring additional training. |
| title | Open-Vocabulary Part-Based Grasping |
| topic | Robotics |
| url | https://arxiv.org/abs/2406.05951 |