A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929287899119616 |
|---|---|
| author | Ingelhag, Nils Munkeby, Jesper van Haastregt, Jonne Varava, Anastasia Welle, Michael C. Kragic, Danica |
| author_facet | Ingelhag, Nils Munkeby, Jesper van Haastregt, Jonne Varava, Anastasia Welle, Michael C. Kragic, Danica |
| contents | In this paper, we build upon two major recent developments in the field, Diffusion Policies for visuomotor manipulation and large pre-trained multimodal foundational models to obtain a robotic skill learning system. The system can obtain new skills via the behavioral cloning approach of visuomotor diffusion policies given teleoperated demonstrations. Foundational models are being used to perform skill selection given the user's prompt in natural language. Before executing a skill the foundational model performs a precondition check given an observation of the workspace. We compare the performance of different foundational models to this end as well as give a detailed experimental evaluation of the skills taught by the user in simulation and the real world. Finally, we showcase the combined system on a challenging food serving scenario in the real world. Videos of all experimental executions, as well as the process of teaching new skills in simulation and the real world, are available on the project's website. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16730 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Models Ingelhag, Nils Munkeby, Jesper van Haastregt, Jonne Varava, Anastasia Welle, Michael C. Kragic, Danica Robotics In this paper, we build upon two major recent developments in the field, Diffusion Policies for visuomotor manipulation and large pre-trained multimodal foundational models to obtain a robotic skill learning system. The system can obtain new skills via the behavioral cloning approach of visuomotor diffusion policies given teleoperated demonstrations. Foundational models are being used to perform skill selection given the user's prompt in natural language. Before executing a skill the foundational model performs a precondition check given an observation of the workspace. We compare the performance of different foundational models to this end as well as give a detailed experimental evaluation of the skills taught by the user in simulation and the real world. Finally, we showcase the combined system on a challenging food serving scenario in the real world. Videos of all experimental executions, as well as the process of teaching new skills in simulation and the real world, are available on the project's website. |
| title | A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Models |
| topic | Robotics |
| url | https://arxiv.org/abs/2403.16730 |