Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.11518 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912033217183744 |
|---|---|
| author | Jiang, Chen Luo, Allie Jagersand, Martin |
| author_facet | Jiang, Chen Luo, Allie Jagersand, Martin |
| contents | In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot's perception module. First, we propose CLIPU$^2$Net, a lightweight referring image segmentation model designed for fine-grain boundary and structure segmentation from language expressions. Then, we deploy the model in an eye-in-hand visual servoing system to enact robot control in the real world. The key to our system is the representation of salient visual information as geometric constraints, linking the robot's visual perception to actionable commands. Experimental results on 46 real-world robot manipulation tasks demonstrate that our method outperforms traditional visual servoing methods relying on labor-intensive feature annotations, excels in fine-grain referring image segmentation with a compact decoder size of 6.6 MB, and supports robot control across diverse contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11518 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints Jiang, Chen Luo, Allie Jagersand, Martin Robotics Computer Vision and Pattern Recognition In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot's perception module. First, we propose CLIPU$^2$Net, a lightweight referring image segmentation model designed for fine-grain boundary and structure segmentation from language expressions. Then, we deploy the model in an eye-in-hand visual servoing system to enact robot control in the real world. The key to our system is the representation of salient visual information as geometric constraints, linking the robot's visual perception to actionable commands. Experimental results on 46 real-world robot manipulation tasks demonstrate that our method outperforms traditional visual servoing methods relying on labor-intensive feature annotations, excels in fine-grain referring image segmentation with a compact decoder size of 6.6 MB, and supports robot control across diverse contexts. |
| title | Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.11518 |