MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908610372567040 |
|---|---|
| author | Wang, Huaxiaoyue Kedia, Kushal Ren, Juntao Abdullah, Rahma Bhardwaj, Atiksh Chao, Angela Chen, Kelly Y Chin, Nathaniel Dan, Prithwish Fan, Xinyi Gonzalez-Pumariega, Gonzalo Kompella, Aditya Pace, Maximus Adrian Sharma, Yash Sun, Xiangwan Sunkara, Neha Choudhury, Sanjiban |
| author_facet | Wang, Huaxiaoyue Kedia, Kushal Ren, Juntao Abdullah, Rahma Bhardwaj, Atiksh Chao, Angela Chen, Kelly Y Chin, Nathaniel Dan, Prithwish Fan, Xinyi Gonzalez-Pumariega, Gonzalo Kompella, Aditya Pace, Maximus Adrian Sharma, Yash Sun, Xiangwan Sunkara, Neha Choudhury, Sanjiban |
| contents | We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. MOSAIC employs modularity at several levels: it leverages multiple large-scale pre-trained models for high-level tasks like language and image recognition, while using streamlined modules designed for low-level task-specific control. This decomposition allows us to reap the complementary benefits of foundation models as well as precise, more specialized models. Pieced together, our system is able to scale to complex tasks that involve coordinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limitations of the current system, and open challenges in this domain. The project's website is at https://portal-cornell.github.io/MOSAIC/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18796 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking Wang, Huaxiaoyue Kedia, Kushal Ren, Juntao Abdullah, Rahma Bhardwaj, Atiksh Chao, Angela Chen, Kelly Y Chin, Nathaniel Dan, Prithwish Fan, Xinyi Gonzalez-Pumariega, Gonzalo Kompella, Aditya Pace, Maximus Adrian Sharma, Yash Sun, Xiangwan Sunkara, Neha Choudhury, Sanjiban Robotics We present MOSAIC, a modular architecture for coordinating multiple robots to (a) interact with users using natural language and (b) manipulate an open vocabulary of everyday objects. MOSAIC employs modularity at several levels: it leverages multiple large-scale pre-trained models for high-level tasks like language and image recognition, while using streamlined modules designed for low-level task-specific control. This decomposition allows us to reap the complementary benefits of foundation models as well as precise, more specialized models. Pieced together, our system is able to scale to complex tasks that involve coordinating multiple robots and humans. First, we unit-test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We then extensively evaluate MOSAIC with 60 end-to-end trials. We discuss crucial design decisions, limitations of the current system, and open challenges in this domain. The project's website is at https://portal-cornell.github.io/MOSAIC/ |
| title | MOSAIC: Modular Foundation Models for Assistive and Interactive Cooking |
| topic | Robotics |
| url | https://arxiv.org/abs/2402.18796 |