Grounding Multimodal Large Language Models in Actions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912148799619072 |
|---|---|
| author | Szot, Andrew Mazoure, Bogdan Agrawal, Harsh Hjelm, Devon Kira, Zsolt Toshev, Alexander |
| author_facet | Szot, Andrew Mazoure, Bogdan Agrawal, Harsh Hjelm, Devon Kira, Zsolt Toshev, Alexander |
| contents | Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 114 embodied tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07904 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Grounding Multimodal Large Language Models in Actions Szot, Andrew Mazoure, Bogdan Agrawal, Harsh Hjelm, Devon Kira, Zsolt Toshev, Alexander Machine Learning Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 114 embodied tasks. |
| title | Grounding Multimodal Large Language Models in Actions |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.07904 |