Language-Conditioned Robotic Manipulation with Fast and Slow Thinking
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914662036013056 |
|---|---|
| author | Zhu, Minjie Zhu, Yichen Li, Jinming Wen, Junjie Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian |
| author_facet | Zhu, Minjie Zhu, Yichen Li, Jinming Wen, Junjie Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian |
| contents | The language-conditioned robotic manipulation aims to transfer natural language instructions into executable actions, from simple pick-and-place to tasks requiring intent recognition and visual reasoning. Inspired by the dual process theory in cognitive science, which suggests two parallel systems of fast and slow thinking in human decision-making, we introduce Robotics with Fast and Slow Thinking (RFST), a framework that mimics human cognitive architecture to classify tasks and makes decisions on two systems based on instruction types. Our RFST consists of two key components: 1) an instruction discriminator to determine which system should be activated based on the current user instruction, and 2) a slow-thinking system that is comprised of a fine-tuned vision language model aligned with the policy networks, which allows the robot to recognize user intention or perform reasoning tasks. To assess our methodology, we built a dataset featuring real-world trajectories, capturing actions ranging from spontaneous impulses to tasks requiring deliberate contemplation. Our results, both in simulation and real-world scenarios, confirm that our approach adeptly manages intricate tasks that demand intent recognition and reasoning. The project is available at https://jlm-z.github.io/RSFT/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04181 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language-Conditioned Robotic Manipulation with Fast and Slow Thinking Zhu, Minjie Zhu, Yichen Li, Jinming Wen, Junjie Xu, Zhiyuan Che, Zhengping Shen, Chaomin Peng, Yaxin Liu, Dong Feng, Feifei Tang, Jian Robotics Computer Vision and Pattern Recognition The language-conditioned robotic manipulation aims to transfer natural language instructions into executable actions, from simple pick-and-place to tasks requiring intent recognition and visual reasoning. Inspired by the dual process theory in cognitive science, which suggests two parallel systems of fast and slow thinking in human decision-making, we introduce Robotics with Fast and Slow Thinking (RFST), a framework that mimics human cognitive architecture to classify tasks and makes decisions on two systems based on instruction types. Our RFST consists of two key components: 1) an instruction discriminator to determine which system should be activated based on the current user instruction, and 2) a slow-thinking system that is comprised of a fine-tuned vision language model aligned with the policy networks, which allows the robot to recognize user intention or perform reasoning tasks. To assess our methodology, we built a dataset featuring real-world trajectories, capturing actions ranging from spontaneous impulses to tasks requiring deliberate contemplation. Our results, both in simulation and real-world scenarios, confirm that our approach adeptly manages intricate tasks that demand intent recognition and reasoning. The project is available at https://jlm-z.github.io/RSFT/ |
| title | Language-Conditioned Robotic Manipulation with Fast and Slow Thinking |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.04181 |