Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2504.01959 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915223968940032 |
|---|---|
| author | Shan, Dandan Mo, Kaichun Yang, Wei Chao, Yu-Wei Fouhey, David Fox, Dieter Mousavian, Arsalan |
| author_facet | Shan, Dandan Mo, Kaichun Yang, Wei Chao, Yu-Wei Fouhey, David Fox, Dieter Mousavian, Arsalan |
| contents | The majority of modern robot learning methods focus on learning a set of pre-defined tasks with limited or no generalization to new tasks. Extending the robot skillset to novel tasks involves gathering an extensive amount of training data for additional tasks. In this paper, we address the problem of teaching new tasks to robots using human demonstration videos for repetitive tasks (e.g., packing). This task requires understanding the human video to identify which object is being manipulated (the pick object) and where it is being placed (the placement slot). In addition, it needs to re-identify the pick object and the placement slots during inference along with the relative poses to enable robot execution of the task. To tackle this, we propose SLeRP, a modular system that leverages several advanced visual foundation models and a novel slot-level placement detector Slot-Net, eliminating the need for expensive video demonstrations for training. We evaluate our system using a new benchmark of real-world videos. The evaluation results show that SLeRP outperforms several baselines and can be deployed on a real robot. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01959 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Slot-Level Robotic Placement via Visual Imitation from Single Human Video Shan, Dandan Mo, Kaichun Yang, Wei Chao, Yu-Wei Fouhey, David Fox, Dieter Mousavian, Arsalan Robotics Computer Vision and Pattern Recognition The majority of modern robot learning methods focus on learning a set of pre-defined tasks with limited or no generalization to new tasks. Extending the robot skillset to novel tasks involves gathering an extensive amount of training data for additional tasks. In this paper, we address the problem of teaching new tasks to robots using human demonstration videos for repetitive tasks (e.g., packing). This task requires understanding the human video to identify which object is being manipulated (the pick object) and where it is being placed (the placement slot). In addition, it needs to re-identify the pick object and the placement slots during inference along with the relative poses to enable robot execution of the task. To tackle this, we propose SLeRP, a modular system that leverages several advanced visual foundation models and a novel slot-level placement detector Slot-Net, eliminating the need for expensive video demonstrations for training. We evaluate our system using a new benchmark of real-world videos. The evaluation results show that SLeRP outperforms several baselines and can be deployed on a real robot. |
| title | Slot-Level Robotic Placement via Visual Imitation from Single Human Video |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.01959 |