Generative Multimodal Models are In-Context Learners
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910437663047680 |
|---|---|
| author | Sun, Quan Cui, Yufeng Zhang, Xiaosong Zhang, Fan Yu, Qiying Luo, Zhengxiong Wang, Yueze Rao, Yongming Liu, Jingjing Huang, Tiejun Wang, Xinlong |
| author_facet | Sun, Quan Cui, Yufeng Zhang, Xiaosong Zhang, Fan Yu, Qiying Luo, Zhengxiong Wang, Yueze Rao, Yongming Liu, Jingjing Huang, Tiejun Wang, Xinlong |
| contents | The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_13286 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Generative Multimodal Models are In-Context Learners Sun, Quan Cui, Yufeng Zhang, Xiaosong Zhang, Fan Yu, Qiying Luo, Zhengxiong Wang, Yueze Rao, Yongming Liu, Jingjing Huang, Tiejun Wang, Xinlong Computer Vision and Pattern Recognition The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research. |
| title | Generative Multimodal Models are In-Context Learners |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.13286 |